2026-02-25T16:29:29.978 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T16:29:29.996 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T16:29:30.071 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_16:24:30-rados:standalone-wip-mon-colocate-distro-default-trial/70446 branch: wip-mon-colocate description: rados:standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} email: null first_in_suite: false flavor: default job_id: '70446' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: emzhang-2026-02-25_16:24:30-rados:standalone-wip-mon-colocate-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mon-colocate ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd workunit: branch: wip-mon-colocate sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd owner: scheduled_emzhang@soko04.front.sepia.ceph.com priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9800 sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-mon-colocate suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd targets: trial129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAAaNxdQB8JZhsxQxysYI4onwK0YQhe6GtNFtKRlhiCD7rFy/9Qzcz1WFnNWA1kNVChbLNacS3NEk85I1qXWI+A= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_16:24:30 tube: trial user: emzhang verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T16:29:30.071 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa; will attempt to use it 2026-02-25T16:29:30.072 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa/tasks 2026-02-25T16:29:30.072 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T16:29:30.073 INFO:teuthology.task.internal:Checking packages... 2026-02-25T16:29:30.073 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd' 2026-02-25T16:29:30.073 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T16:29:30.073 INFO:teuthology.packaging:ref: None 2026-02-25T16:29:30.073 INFO:teuthology.packaging:tag: None 2026-02-25T16:29:30.074 INFO:teuthology.packaging:branch: wip-mon-colocate 2026-02-25T16:29:30.074 INFO:teuthology.packaging:sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T16:29:30.074 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mon-colocate 2026-02-25T16:29:30.192 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5305.g0886691e 2026-02-25T16:29:30.193 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T16:29:30.205 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T16:29:30.205 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T16:29:30.219 INFO:teuthology.task.internal:Saving configuration 2026-02-25T16:29:30.227 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T16:29:30.243 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T16:29:30.330 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial129.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/emzhang-2026-02-25_16:24:30-rados:standalone-wip-mon-colocate-distro-default-trial/70446', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 16:25:57.973159', 'locked_by': 'scheduled_emzhang@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAAaNxdQB8JZhsxQxysYI4onwK0YQhe6GtNFtKRlhiCD7rFy/9Qzcz1WFnNWA1kNVChbLNacS3NEk85I1qXWI+A='} 2026-02-25T16:29:30.330 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T16:29:30.341 INFO:teuthology.task.internal:roles: ubuntu@trial129.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-25T16:29:30.341 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T16:29:30.423 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7b46034711c0>, signals=[15]) 2026-02-25T16:29:30.423 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T16:29:30.434 INFO:teuthology.task.internal:Opening connections... 2026-02-25T16:29:30.434 DEBUG:teuthology.task.internal:connecting to ubuntu@trial129.front.sepia.ceph.com 2026-02-25T16:29:30.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T16:29:30.499 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T16:29:30.510 DEBUG:teuthology.orchestra.run.trial129:> uname -m 2026-02-25T16:29:30.580 INFO:teuthology.orchestra.run.trial129.stdout:x86_64 2026-02-25T16:29:30.580 DEBUG:teuthology.orchestra.run.trial129:> cat /etc/os-release 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:NAME="CentOS Stream" 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:VERSION="9" 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:ID="centos" 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:ID_LIKE="rhel fedora" 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:VERSION_ID="9" 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:PLATFORM_ID="platform:el9" 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T16:29:30.636 INFO:teuthology.orchestra.run.trial129.stdout:ANSI_COLOR="0;31" 2026-02-25T16:29:30.637 INFO:teuthology.orchestra.run.trial129.stdout:LOGO="fedora-logo-icon" 2026-02-25T16:29:30.637 INFO:teuthology.orchestra.run.trial129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T16:29:30.637 INFO:teuthology.orchestra.run.trial129.stdout:HOME_URL="https://centos.org/" 2026-02-25T16:29:30.637 INFO:teuthology.orchestra.run.trial129.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T16:29:30.637 INFO:teuthology.orchestra.run.trial129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T16:29:30.637 INFO:teuthology.orchestra.run.trial129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T16:29:30.637 INFO:teuthology.lock.ops:Updating trial129.front.sepia.ceph.com on lock server 2026-02-25T16:29:30.717 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T16:29:30.730 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T16:29:30.743 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T16:29:30.743 DEBUG:teuthology.orchestra.run.trial129:> test '!' -e /home/ubuntu/cephtest 2026-02-25T16:29:30.758 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T16:29:30.769 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T16:29:30.769 DEBUG:teuthology.orchestra.run.trial129:> test -z $(ls -A /var/lib/ceph) 2026-02-25T16:29:30.815 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T16:29:30.900 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T16:29:30.919 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T16:29:30.919 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T16:29:30.920 DEBUG:teuthology.orchestra.run.trial129:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T16:29:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T16:29:30.934 DEBUG:teuthology.orchestra.run.trial129:> uname -r 2026-02-25T16:29:30.989 INFO:teuthology.orchestra.run.trial129.stdout:5.14.0-681.el9.x86_64 2026-02-25T16:29:30.989 INFO:teuthology.task.kernel:Running kernel on trial129: 5.14.0-681.el9.x86_64 2026-02-25T16:29:30.990 DEBUG:teuthology.orchestra.run.trial129:> sudo yum install -y kernel 2026-02-25T16:29:39.669 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.9 MB 00:08 2026-02-25T16:29:43.436 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 27 MB 00:03 2026-02-25T16:29:46.796 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 8.0 MB 00:00 2026-02-25T16:29:48.450 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 20 kB 00:00 2026-02-25T16:29:50.115 INFO:teuthology.orchestra.run.trial129.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2026-02-25T16:29:53.298 INFO:teuthology.orchestra.run.trial129.stdout:lab-extras 1.8 MB/s | 50 kB 00:00 2026-02-25T16:29:54.303 INFO:teuthology.orchestra.run.trial129.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T16:29:54.304 INFO:teuthology.orchestra.run.trial129.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T16:29:54.324 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout: Package Architecture Version Repository Size 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout:Installing dependencies: 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T16:29:54.325 INFO:teuthology.orchestra.run.trial129.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout:Install 4 Packages 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 89 M 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 135 M 2026-02-25T16:29:54.326 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:29:54.442 INFO:teuthology.orchestra.run.trial129.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 30 MB/s | 901 kB 00:00 2026-02-25T16:29:54.593 INFO:teuthology.orchestra.run.trial129.stdout:(2/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 222 MB/s | 40 MB 00:00 2026-02-25T16:29:54.618 INFO:teuthology.orchestra.run.trial129.stdout:(3/4): kernel-core-5.14.0-686.el9.x86_64.rpm 84 MB/s | 17 MB 00:00 2026-02-25T16:29:54.668 INFO:teuthology.orchestra.run.trial129.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 138 MB/s | 31 MB 00:00 2026-02-25T16:29:54.668 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:29:54.668 INFO:teuthology.orchestra.run.trial129.stdout:Total 260 MB/s | 89 MB 00:00 2026-02-25T16:29:54.857 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:29:55.060 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:29:55.060 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:29:55.445 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:29:55.445 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:29:56.553 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:29:56.842 INFO:teuthology.orchestra.run.trial129.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T16:29:56.864 INFO:teuthology.orchestra.run.trial129.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T16:29:59.430 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T16:29:59.463 INFO:teuthology.orchestra.run.trial129.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T16:30:02.510 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T16:30:02.515 INFO:teuthology.orchestra.run.trial129.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T16:30:05.574 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T16:30:23.043 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T16:30:23.064 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T16:30:23.252 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T16:30:23.252 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T16:30:23.252 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T16:30:23.252 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T16:30:23.309 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T16:30:23.309 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:30:23.309 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:30:23.309 INFO:teuthology.orchestra.run.trial129.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T16:30:23.309 INFO:teuthology.orchestra.run.trial129.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T16:30:23.310 INFO:teuthology.orchestra.run.trial129.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T16:30:23.310 INFO:teuthology.orchestra.run.trial129.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T16:30:23.310 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:30:23.310 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:30:23.369 DEBUG:teuthology.orchestra.run.trial129:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T16:30:23.398 INFO:teuthology.orchestra.run.trial129.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T16:30:23.399 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T16:30:23.399 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T16:30:23.399 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T16:30:23.399 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T16:30:23.399 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T16:30:23.399 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial129.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T16:30:23.399 DEBUG:teuthology.orchestra.run.trial129:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T16:30:23.467 INFO:teuthology.orchestra.run.trial129.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T16:30:23.468 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T16:30:23.468 DEBUG:teuthology.orchestra.run.trial129:> sudo rpm -qi grub2-tools 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Name : grub2-tools 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Epoch : 1 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Version : 2.06 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Release : 120.el9 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Architecture: x86_64 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Group : Unspecified 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:Size : 9108611 2026-02-25T16:30:23.537 INFO:teuthology.orchestra.run.trial129.stdout:License : GPLv3+ 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Packager : builder@centos.org 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Vendor : CentOS 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Summary : Support tools for GRUB. 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:Description : 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:30:23.538 INFO:teuthology.orchestra.run.trial129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T16:30:23.539 INFO:teuthology.orchestra.run.trial129.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T16:30:23.539 INFO:teuthology.orchestra.run.trial129.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T16:30:23.539 INFO:teuthology.orchestra.run.trial129.stdout:hardware devices. 2026-02-25T16:30:23.539 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:30:23.539 INFO:teuthology.orchestra.run.trial129.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T16:30:23.540 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T16:30:23.540 INFO:teuthology.task.kernel:Updating grub on trial129 to boot 5.14.0-686.el9.x86_64 2026-02-25T16:30:23.540 DEBUG:teuthology.orchestra.run.trial129:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T16:30:23.606 INFO:teuthology.orchestra.run.trial129.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T16:30:23.607 DEBUG:teuthology.orchestra.run.trial129:> sudo command -v grubby && echo yes || echo no 2026-02-25T16:30:23.671 INFO:teuthology.orchestra.run.trial129.stdout:/sbin/grubby 2026-02-25T16:30:23.672 INFO:teuthology.orchestra.run.trial129.stdout:yes 2026-02-25T16:30:23.672 DEBUG:teuthology.orchestra.run.trial129:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T16:30:23.737 INFO:teuthology.orchestra.run.trial129.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T16:30:23.737 DEBUG:teuthology.orchestra.run.trial129:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T16:30:23.810 DEBUG:teuthology.orchestra.run.trial129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T16:30:24.018 INFO:teuthology.orchestra.run.trial129.stderr:Generating grub configuration file ... 2026-02-25T16:30:24.085 INFO:teuthology.orchestra.run.trial129.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T16:30:24.092 INFO:teuthology.orchestra.run.trial129.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T16:30:24.344 INFO:teuthology.orchestra.run.trial129.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T16:30:24.350 INFO:teuthology.orchestra.run.trial129.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T16:30:24.384 INFO:teuthology.orchestra.run.trial129.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T16:30:24.391 INFO:teuthology.orchestra.run.trial129.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T16:30:24.422 INFO:teuthology.orchestra.run.trial129.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T16:30:24.429 INFO:teuthology.orchestra.run.trial129.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T16:30:24.453 INFO:teuthology.orchestra.run.trial129.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T16:30:24.459 INFO:teuthology.orchestra.run.trial129.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T16:30:24.545 INFO:teuthology.orchestra.run.trial129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T16:30:24.554 INFO:teuthology.orchestra.run.trial129.stderr:done 2026-02-25T16:30:24.555 DEBUG:teuthology.orchestra.run.trial129:> test -d /sys/firmware/efi 2026-02-25T16:30:24.571 DEBUG:teuthology.orchestra.run.trial129:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T16:30:24.637 INFO:teuthology.orchestra.run.trial129.stdout:/boot/efi/EFI/BOOT 2026-02-25T16:30:24.638 INFO:teuthology.orchestra.run.trial129.stdout:/boot/efi/EFI/centos 2026-02-25T16:30:24.639 DEBUG:teuthology.orchestra.run.trial129:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T16:30:24.704 INFO:teuthology.orchestra.run.trial129.stdout:no 2026-02-25T16:30:24.705 DEBUG:teuthology.orchestra.run.trial129:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T16:30:24.768 INFO:teuthology.orchestra.run.trial129.stdout:yes 2026-02-25T16:30:24.769 DEBUG:teuthology.orchestra.run.trial129:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T16:30:24.834 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial129 2026-02-25T16:30:24.834 DEBUG:teuthology.orchestra.run.trial129:> sudo bash -c ' 2026-02-25T16:30:24.834 DEBUG:teuthology.orchestra.run.trial129:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T16:30:24.834 DEBUG:teuthology.orchestra.run.trial129:> ' 2026-02-25T16:30:54.877 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T16:30:54.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial129.front.sepia.ceph.com' 2026-02-25T16:30:54.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T16:31:09.184 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.129 2026-02-25T16:31:18.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial129.front.sepia.ceph.com' 2026-02-25T16:31:18.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T16:31:54.373 DEBUG:teuthology.orchestra.run.trial129:> true 2026-02-25T16:31:54.543 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial129.front.sepia.ceph.com' 2026-02-25T16:31:54.543 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T16:31:54.544 DEBUG:teuthology.orchestra.run.trial129:> uname -r 2026-02-25T16:31:54.599 INFO:teuthology.orchestra.run.trial129.stdout:5.14.0-686.el9.x86_64 2026-02-25T16:31:54.599 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T16:31:54.600 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T16:31:54.600 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T16:31:55.601 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T16:31:55.601 DEBUG:teuthology.orchestra.run.trial129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T16:31:55.632 INFO:teuthology.orchestra.run.trial129.stdout:ttyS1 2026-02-25T16:31:55.651 DEBUG:teuthology.parallel:result is None 2026-02-25T16:31:55.651 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T16:31:55.664 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T16:31:55.664 DEBUG:teuthology.orchestra.run.trial129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T16:31:55.708 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T16:31:55.721 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T16:31:55.736 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T16:31:55.736 DEBUG:teuthology.orchestra.run.trial129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T16:31:55.770 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T16:31:55.782 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T16:31:55.782 DEBUG:teuthology.orchestra.run.trial129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T16:31:55.828 INFO:teuthology.orchestra.run.trial129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:31:55.837 INFO:teuthology.orchestra.run.trial129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:31:55.839 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T16:31:55.852 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T16:31:55.852 DEBUG:teuthology.orchestra.run.trial129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T16:31:55.905 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T16:31:55.925 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T16:31:55.925 DEBUG:teuthology.orchestra.run.trial129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T16:31:55.961 DEBUG:teuthology.orchestra.run.trial129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T16:31:56.122 DEBUG:teuthology.orchestra.run.trial129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T16:31:56.148 DEBUG:teuthology.orchestra.run.trial129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T16:31:56.301 DEBUG:teuthology.orchestra.run.trial129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T16:31:56.326 DEBUG:teuthology.orchestra.run.trial129:> set -ex 2026-02-25T16:31:56.326 DEBUG:teuthology.orchestra.run.trial129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T16:31:56.390 DEBUG:teuthology.orchestra.run.trial129:> sudo service rsyslog restart 2026-02-25T16:31:56.457 INFO:teuthology.orchestra.run.trial129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T16:31:56.812 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T16:31:56.826 INFO:teuthology.task.internal:Starting timer... 2026-02-25T16:31:56.826 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T16:31:56.852 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T16:31:56.876 DEBUG:teuthology.orchestra.run.trial129:> sudo service auditd rotate 2026-02-25T16:31:56.908 INFO:teuthology.orchestra.run.trial129.stdout:Rotating logs: 2026-02-25T16:31:56.909 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T16:31:56.910 DEBUG:teuthology.orchestra.run.trial129:> /usr/sbin/getenforce 2026-02-25T16:31:56.967 INFO:teuthology.orchestra.run.trial129.stdout:Permissive 2026-02-25T16:31:56.968 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial129.front.sepia.ceph.com': 'permissive'} 2026-02-25T16:31:56.968 DEBUG:teuthology.orchestra.run.trial129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T16:31:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T16:31:57.033 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T16:31:57.034 DEBUG:teuthology.orchestra.run.trial129:> sudo /usr/sbin/setenforce permissive 2026-02-25T16:31:57.097 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T16:31:57.113 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T16:31:57.126 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T16:31:57.128 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T16:34:43.894 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial129.front.sepia.ceph.com')] 2026-02-25T16:34:43.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial129.front.sepia.ceph.com' 2026-02-25T16:34:43.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T16:34:44.007 DEBUG:teuthology.orchestra.run.trial129:> true 2026-02-25T16:34:44.088 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial129.front.sepia.ceph.com' 2026-02-25T16:34:44.088 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T16:34:44.098 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T16:34:44.098 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T16:34:44.098 DEBUG:teuthology.orchestra.run.trial129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T16:34:44.156 INFO:teuthology.orchestra.run.trial129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T16:34:44.168 INFO:teuthology.orchestra.run.trial129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T16:34:44.190 INFO:teuthology.orchestra.run.trial129.stderr:sudo: ntpd: command not found 2026-02-25T16:34:44.200 INFO:teuthology.orchestra.run.trial129.stdout:506 Cannot talk to daemon 2026-02-25T16:34:44.211 INFO:teuthology.orchestra.run.trial129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T16:34:44.223 INFO:teuthology.orchestra.run.trial129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T16:34:44.275 INFO:teuthology.orchestra.run.trial129.stderr:bash: line 1: ntpq: command not found 2026-02-25T16:34:44.310 INFO:teuthology.orchestra.run.trial129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T16:34:44.310 INFO:teuthology.orchestra.run.trial129.stdout:=============================================================================== 2026-02-25T16:34:44.310 INFO:teuthology.orchestra.run.trial129.stdout:^? 69-172-133-130.planet.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T16:34:44.310 INFO:teuthology.orchestra.run.trial129.stdout:^? ip229.ip-51-81-226.us 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T16:34:44.311 INFO:teuthology.orchestra.run.trial129.stdout:^? connected.by.freedominte> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T16:34:44.311 INFO:teuthology.orchestra.run.trial129.stdout:^? 204.2.134.173 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T16:34:44.311 INFO:teuthology.run_tasks:Running task install... 2026-02-25T16:34:44.325 DEBUG:teuthology.task.install:project ceph 2026-02-25T16:34:44.325 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'}} 2026-02-25T16:34:44.326 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'} 2026-02-25T16:34:44.326 INFO:teuthology.task.install:Using flavor: default 2026-02-25T16:34:44.335 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T16:34:44.335 INFO:teuthology.task.install:extra packages: [] 2026-02-25T16:34:44.335 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd', 'tag': None, 'wait_for_package': False} 2026-02-25T16:34:44.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T16:34:44.508 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/centos/9/flavors/default/ 2026-02-25T16:34:44.508 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5305.g0886691e 2026-02-25T16:34:44.626 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T16:34:44.626 DEBUG:teuthology.orchestra.run.trial129:> set -ex 2026-02-25T16:34:44.626 DEBUG:teuthology.orchestra.run.trial129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T16:34:44.651 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T16:34:44.652 DEBUG:teuthology.orchestra.run.trial129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T16:34:44.717 DEBUG:teuthology.orchestra.run.trial129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T16:34:44.791 DEBUG:teuthology.orchestra.run.trial129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T16:34:44.855 INFO:teuthology.orchestra.run.trial129.stdout:check_obsoletes = 1 2026-02-25T16:34:44.857 DEBUG:teuthology.orchestra.run.trial129:> sudo yum clean all 2026-02-25T16:34:45.005 INFO:teuthology.orchestra.run.trial129.stdout:41 files removed 2026-02-25T16:34:45.019 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-radosgw 2026-02-25T16:34:45.298 INFO:teuthology.orchestra.run.trial129.stdout:ceph packages for x86_64 723 kB/s | 93 kB 00:00 2026-02-25T16:34:45.442 INFO:teuthology.orchestra.run.trial129.stdout:ceph noarch packages 213 kB/s | 24 kB 00:00 2026-02-25T16:34:45.572 INFO:teuthology.orchestra.run.trial129.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-25T16:34:46.575 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.9 MB 00:00 2026-02-25T16:34:50.311 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 27 MB 00:03 2026-02-25T16:34:54.200 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 8.0 MB 00:01 2026-02-25T16:34:55.330 INFO:teuthology.orchestra.run.trial129.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 20 kB 00:00 2026-02-25T16:34:56.557 INFO:teuthology.orchestra.run.trial129.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-25T16:34:59.736 INFO:teuthology.orchestra.run.trial129.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T16:35:00.942 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:00.943 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:00.943 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repository Size 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout: ceph-radosgw x86_64 2:20.3.0-5305.g0886691e.el9 ceph 13 M 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout:Upgrading: 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout: librados2 x86_64 2:20.3.0-5305.g0886691e.el9 ceph 3.5 M 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout: librbd1 x86_64 2:20.3.0-5305.g0886691e.el9 ceph 2.8 M 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout:Installing dependencies: 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout: ceph-base x86_64 2:20.3.0-5305.g0886691e.el9 ceph 5.2 M 2026-02-25T16:35:00.944 INFO:teuthology.orchestra.run.trial129.stdout: ceph-common x86_64 2:20.3.0-5305.g0886691e.el9 ceph 24 M 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: ceph-selinux x86_64 2:20.3.0-5305.g0886691e.el9 ceph 25 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs2 x86_64 2:20.3.0-5305.g0886691e.el9 ceph 910 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: libradosstriper1 x86_64 2:20.3.0-5305.g0886691e.el9 ceph 248 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T16:35:00.945 INFO:teuthology.orchestra.run.trial129.stdout: librgw2 x86_64 2:20.3.0-5305.g0886691e.el9 ceph 6.5 M 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: python3-ceph-argparse x86_64 2:20.3.0-5305.g0886691e.el9 ceph 45 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: python3-ceph-common x86_64 2:20.3.0-5305.g0886691e.el9 ceph 185 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: python3-cephfs x86_64 2:20.3.0-5305.g0886691e.el9 ceph 203 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: python3-rados x86_64 2:20.3.0-5305.g0886691e.el9 ceph 324 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: python3-rbd x86_64 2:20.3.0-5305.g0886691e.el9 ceph 305 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: python3-rgw x86_64 2:20.3.0-5305.g0886691e.el9 ceph 100 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T16:35:00.946 INFO:teuthology.orchestra.run.trial129.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout:Installing weak dependencies: 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs-daemon x86_64 2:20.3.0-5305.g0886691e.el9 ceph 37 k 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5305.g0886691e.el9 ceph 26 k 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout:Install 35 Packages 2026-02-25T16:35:00.947 INFO:teuthology.orchestra.run.trial129.stdout:Upgrade 2 Packages 2026-02-25T16:35:00.948 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:00.948 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 68 M 2026-02-25T16:35:00.948 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:01.932 INFO:teuthology.orchestra.run.trial129.stdout:(1/37): ceph-base-20.3.0-5305.g0886691e.el9.x86 17 MB/s | 5.2 MB 00:00 2026-02-25T16:35:01.969 INFO:teuthology.orchestra.run.trial129.stdout:(2/37): ceph-selinux-20.3.0-5305.g0886691e.el9. 695 kB/s | 25 kB 00:00 2026-02-25T16:35:02.008 INFO:teuthology.orchestra.run.trial129.stdout:(3/37): libcephfs-daemon-20.3.0-5305.g0886691e. 945 kB/s | 37 kB 00:00 2026-02-25T16:35:02.049 INFO:teuthology.orchestra.run.trial129.stdout:(4/37): libcephfs-proxy2-20.3.0-5305.g0886691e. 627 kB/s | 26 kB 00:00 2026-02-25T16:35:02.110 INFO:teuthology.orchestra.run.trial129.stdout:(5/37): libcephfs2-20.3.0-5305.g0886691e.el9.x8 15 MB/s | 910 kB 00:00 2026-02-25T16:35:02.160 INFO:teuthology.orchestra.run.trial129.stdout:(6/37): libradosstriper1-20.3.0-5305.g0886691e. 4.8 MB/s | 248 kB 00:00 2026-02-25T16:35:02.392 INFO:teuthology.orchestra.run.trial129.stdout:(7/37): ceph-radosgw-20.3.0-5305.g0886691e.el9. 17 MB/s | 13 MB 00:00 2026-02-25T16:35:02.426 INFO:teuthology.orchestra.run.trial129.stdout:(8/37): python3-ceph-argparse-20.3.0-5305.g0886 1.3 MB/s | 45 kB 00:00 2026-02-25T16:35:02.467 INFO:teuthology.orchestra.run.trial129.stdout:(9/37): python3-ceph-common-20.3.0-5305.g088669 4.5 MB/s | 185 kB 00:00 2026-02-25T16:35:02.479 INFO:teuthology.orchestra.run.trial129.stdout:(10/37): librgw2-20.3.0-5305.g0886691e.el9.x86_ 20 MB/s | 6.5 MB 00:00 2026-02-25T16:35:02.593 INFO:teuthology.orchestra.run.trial129.stdout:(11/37): python3-cephfs-20.3.0-5305.g0886691e.e 1.6 MB/s | 203 kB 00:00 2026-02-25T16:35:02.599 INFO:teuthology.orchestra.run.trial129.stdout:(12/37): python3-rados-20.3.0-5305.g0886691e.el 2.6 MB/s | 324 kB 00:00 2026-02-25T16:35:02.745 INFO:teuthology.orchestra.run.trial129.stdout:(13/37): python3-rbd-20.3.0-5305.g0886691e.el9. 2.0 MB/s | 305 kB 00:00 2026-02-25T16:35:02.746 INFO:teuthology.orchestra.run.trial129.stdout:(14/37): python3-rgw-20.3.0-5305.g0886691e.el9. 682 kB/s | 100 kB 00:00 2026-02-25T16:35:02.797 INFO:teuthology.orchestra.run.trial129.stdout:(15/37): ceph-common-20.3.0-5305.g0886691e.el9. 21 MB/s | 24 MB 00:01 2026-02-25T16:35:02.855 INFO:teuthology.orchestra.run.trial129.stdout:(16/37): mailcap-2.1.49-5.el9.noarch.rpm 303 kB/s | 33 kB 00:00 2026-02-25T16:35:02.858 INFO:teuthology.orchestra.run.trial129.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 1.5 MB/s | 93 kB 00:00 2026-02-25T16:35:02.872 INFO:teuthology.orchestra.run.trial129.stdout:(18/37): nvme-cli-2.16-1.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2026-02-25T16:35:03.432 INFO:teuthology.orchestra.run.trial129.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 180 kB/s | 104 kB 00:00 2026-02-25T16:35:03.518 INFO:teuthology.orchestra.run.trial129.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 528 kB/s | 45 kB 00:00 2026-02-25T16:35:03.525 INFO:teuthology.orchestra.run.trial129.stdout:(21/37): libnbd-1.20.3-4.el9.x86_64.rpm 245 kB/s | 164 kB 00:00 2026-02-25T16:35:03.765 INFO:teuthology.orchestra.run.trial129.stdout:(22/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 179 kB/s | 160 kB 00:00 2026-02-25T16:35:03.927 INFO:teuthology.orchestra.run.trial129.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 728 kB/s | 292 kB 00:00 2026-02-25T16:35:04.037 INFO:teuthology.orchestra.run.trial129.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 338 kB/s | 37 kB 00:00 2026-02-25T16:35:04.146 INFO:teuthology.orchestra.run.trial129.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 607 kB/s | 66 kB 00:00 2026-02-25T16:35:04.151 INFO:teuthology.orchestra.run.trial129.stdout:(26/37): librdkafka-1.6.1-102.el9.x86_64.rpm 1.0 MB/s | 662 kB 00:00 2026-02-25T16:35:04.319 INFO:teuthology.orchestra.run.trial129.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.7 MB/s | 308 kB 00:00 2026-02-25T16:35:04.344 INFO:teuthology.orchestra.run.trial129.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 979 kB/s | 25 kB 00:00 2026-02-25T16:35:04.370 INFO:teuthology.orchestra.run.trial129.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2026-02-25T16:35:04.395 INFO:teuthology.orchestra.run.trial129.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2026-02-25T16:35:04.415 INFO:teuthology.orchestra.run.trial129.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2026-02-25T16:35:04.441 INFO:teuthology.orchestra.run.trial129.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 7.0 MB/s | 191 kB 00:00 2026-02-25T16:35:04.443 INFO:teuthology.orchestra.run.trial129.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 17 MB/s | 838 kB 00:00 2026-02-25T16:35:04.472 INFO:teuthology.orchestra.run.trial129.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 52 MB/s | 1.6 MB 00:00 2026-02-25T16:35:04.518 INFO:teuthology.orchestra.run.trial129.stdout:(35/37): qatlib-25.08.0-2.el9.x86_64.rpm 318 kB/s | 240 kB 00:00 2026-02-25T16:35:04.804 INFO:teuthology.orchestra.run.trial129.stdout:(36/37): librados2-20.3.0-5305.g0886691e.el9.x8 9.7 MB/s | 3.5 MB 00:00 2026-02-25T16:35:04.813 INFO:teuthology.orchestra.run.trial129.stdout:(37/37): librbd1-20.3.0-5305.g0886691e.el9.x86_ 8.3 MB/s | 2.8 MB 00:00 2026-02-25T16:35:04.817 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:04.817 INFO:teuthology.orchestra.run.trial129.stdout:Total 18 MB/s | 68 MB 00:03 2026-02-25T16:35:04.909 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:04.928 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:04.928 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:05.194 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:05.194 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:05.504 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:05.655 INFO:teuthology.orchestra.run.trial129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T16:35:05.658 INFO:teuthology.orchestra.run.trial129.stdout: Upgrading : librados2-2:20.3.0-5305.g0886691e.el9.x86_64 2/39 2026-02-25T16:35:05.710 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: librados2-2:20.3.0-5305.g0886691e.el9.x86_64 2/39 2026-02-25T16:35:05.711 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libcephfs2-2:20.3.0-5305.g0886691e.el9.x86_64 3/39 2026-02-25T16:35:05.724 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: libcephfs2-2:20.3.0-5305.g0886691e.el9.x86_64 3/39 2026-02-25T16:35:05.726 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libcephfs-daemon-2:20.3.0-5305.g0886691e.el9.x86_6 4/39 2026-02-25T16:35:05.727 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libcephfs-proxy2-2:20.3.0-5305.g0886691e.el9.x86_6 5/39 2026-02-25T16:35:05.754 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5305.g0886691e.el9.x86_6 5/39 2026-02-25T16:35:05.756 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-rados-2:20.3.0-5305.g0886691e.el9.x86_64 6/39 2026-02-25T16:35:05.764 INFO:teuthology.orchestra.run.trial129.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T16:35:05.767 INFO:teuthology.orchestra.run.trial129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T16:35:05.770 INFO:teuthology.orchestra.run.trial129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T16:35:05.774 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T16:35:05.785 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-ceph-argparse-2:20.3.0-5305.g0886691e.el9. 11/39 2026-02-25T16:35:05.796 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-cephfs-2:20.3.0-5305.g0886691e.el9.x86_64 12/39 2026-02-25T16:35:05.797 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libradosstriper1-2:20.3.0-5305.g0886691e.el9.x86_6 13/39 2026-02-25T16:35:05.818 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5305.g0886691e.el9.x86_6 13/39 2026-02-25T16:35:05.821 INFO:teuthology.orchestra.run.trial129.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T16:35:05.851 INFO:teuthology.orchestra.run.trial129.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T16:35:05.854 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T16:35:05.862 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T16:35:05.864 INFO:teuthology.orchestra.run.trial129.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T16:35:05.868 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T16:35:05.989 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T16:35:05.990 INFO:teuthology.orchestra.run.trial129.stdout: Upgrading : librbd1-2:20.3.0-5305.g0886691e.el9.x86_64 21/39 2026-02-25T16:35:06.017 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: librbd1-2:20.3.0-5305.g0886691e.el9.x86_64 21/39 2026-02-25T16:35:06.019 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-rbd-2:20.3.0-5305.g0886691e.el9.x86_64 22/39 2026-02-25T16:35:06.025 INFO:teuthology.orchestra.run.trial129.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T16:35:06.306 INFO:teuthology.orchestra.run.trial129.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T16:35:06.307 INFO:teuthology.orchestra.run.trial129.stdout: Installing : librgw2-2:20.3.0-5305.g0886691e.el9.x86_64 25/39 2026-02-25T16:35:06.324 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: librgw2-2:20.3.0-5305.g0886691e.el9.x86_64 25/39 2026-02-25T16:35:06.327 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-rgw-2:20.3.0-5305.g0886691e.el9.x86_64 26/39 2026-02-25T16:35:06.333 INFO:teuthology.orchestra.run.trial129.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T16:35:06.336 INFO:teuthology.orchestra.run.trial129.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T16:35:06.336 INFO:teuthology.orchestra.run.trial129.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T16:35:06.354 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T16:35:06.469 INFO:teuthology.orchestra.run.trial129.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T16:35:06.471 INFO:teuthology.orchestra.run.trial129.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T16:35:06.740 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T16:35:06.741 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T16:35:06.741 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:06.783 INFO:teuthology.orchestra.run.trial129.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T16:35:06.786 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-ceph-common-2:20.3.0-5305.g0886691e.el9.x8 33/39 2026-02-25T16:35:07.877 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-common-2:20.3.0-5305.g0886691e.el9.x86_64 34/39 2026-02-25T16:35:07.880 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-common-2:20.3.0-5305.g0886691e.el9.x86_64 34/39 2026-02-25T16:35:08.139 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-common-2:20.3.0-5305.g0886691e.el9.x86_64 34/39 2026-02-25T16:35:08.142 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-base-2:20.3.0-5305.g0886691e.el9.x86_64 35/39 2026-02-25T16:35:08.179 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-base-2:20.3.0-5305.g0886691e.el9.x86_64 35/39 2026-02-25T16:35:08.179 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T16:35:08.179 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T16:35:08.179 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:08.182 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-selinux-2:20.3.0-5305.g0886691e.el9.x86_64 36/39 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5305.g0886691e.el9.x86_64 36/39 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout:skipping the directory /sys 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout:skipping the directory /proc 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout:skipping the directory /mnt 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout:skipping the directory /var/tmp 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout:skipping the directory /home 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout:skipping the directory /root 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout:skipping the directory /tmp 2026-02-25T16:35:12.689 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:13.266 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-radosgw-2:20.3.0-5305.g0886691e.el9.x86_64 37/39 2026-02-25T16:35:13.289 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5305.g0886691e.el9.x86_64 37/39 2026-02-25T16:35:13.290 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:13.290 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T16:35:13.290 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T16:35:13.290 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T16:35:13.290 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:13.291 INFO:teuthology.orchestra.run.trial129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T16:35:13.304 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T16:35:13.305 INFO:teuthology.orchestra.run.trial129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T16:35:13.362 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T16:35:13.362 INFO:teuthology.orchestra.run.trial129.stdout:Creating group 'qat' with GID 992. 2026-02-25T16:35:13.362 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:13.652 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-base-2:20.3.0-5305.g0886691e.el9.x86_64 1/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-common-2:20.3.0-5305.g0886691e.el9.x86_64 2/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-radosgw-2:20.3.0-5305.g0886691e.el9.x86_64 3/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-selinux-2:20.3.0-5305.g0886691e.el9.x86_64 4/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libcephfs-daemon-2:20.3.0-5305.g0886691e.el9.x86_6 5/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5305.g0886691e.el9.x86_6 6/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libcephfs2-2:20.3.0-5305.g0886691e.el9.x86_64 7/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libradosstriper1-2:20.3.0-5305.g0886691e.el9.x86_6 8/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librgw2-2:20.3.0-5305.g0886691e.el9.x86_64 9/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5305.g0886691e.el9. 10/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-ceph-common-2:20.3.0-5305.g0886691e.el9.x8 11/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-cephfs-2:20.3.0-5305.g0886691e.el9.x86_64 12/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-rados-2:20.3.0-5305.g0886691e.el9.x86_64 13/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-rbd-2:20.3.0-5305.g0886691e.el9.x86_64 14/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-rgw-2:20.3.0-5305.g0886691e.el9.x86_64 15/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librados2-2:20.3.0-5305.g0886691e.el9.x86_64 36/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T16:35:13.653 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librbd1-2:20.3.0-5305.g0886691e.el9.x86_64 38/39 2026-02-25T16:35:13.721 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T16:35:13.721 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:13.721 INFO:teuthology.orchestra.run.trial129.stdout:Upgraded: 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: librados2-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: librbd1-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: ceph-base-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: ceph-common-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: ceph-radosgw-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: ceph-selinux-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T16:35:13.722 INFO:teuthology.orchestra.run.trial129.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs-daemon-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs-proxy2-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs2-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: libradosstriper1-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: librgw2-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: python3-ceph-argparse-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.723 INFO:teuthology.orchestra.run.trial129.stdout: python3-ceph-common-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: python3-cephfs-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: python3-rados-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: python3-rbd-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: python3-rgw-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:13.724 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:13.762 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-test 2026-02-25T16:35:14.074 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:15 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:14.157 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:14.158 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:14.158 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repository Size 2026-02-25T16:35:14.158 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:14.158 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:14.158 INFO:teuthology.orchestra.run.trial129.stdout: ceph-test x86_64 2:20.3.0-5305.g0886691e.el9 ceph 78 M 2026-02-25T16:35:14.158 INFO:teuthology.orchestra.run.trial129.stdout:Installing dependencies: 2026-02-25T16:35:14.158 INFO:teuthology.orchestra.run.trial129.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: libcephsqlite x86_64 2:20.3.0-5305.g0886691e.el9 ceph 159 k 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:14.159 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:14.160 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:14.160 INFO:teuthology.orchestra.run.trial129.stdout:Install 10 Packages 2026-02-25T16:35:14.160 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:14.160 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 82 M 2026-02-25T16:35:14.160 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 339 M 2026-02-25T16:35:14.160 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:14.972 INFO:teuthology.orchestra.run.trial129.stdout:(1/10): libcephsqlite-20.3.0-5305.g0886691e.el9 1.6 MB/s | 159 kB 00:00 2026-02-25T16:35:15.374 INFO:teuthology.orchestra.run.trial129.stdout:(2/10): libatomic-11.5.0-14.el9.x86_64.rpm 57 kB/s | 23 kB 00:00 2026-02-25T16:35:15.577 INFO:teuthology.orchestra.run.trial129.stdout:(3/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 464 kB/s | 94 kB 00:00 2026-02-25T16:35:15.670 INFO:teuthology.orchestra.run.trial129.stdout:(4/10): libxslt-1.1.34-12.el9.x86_64.rpm 2.5 MB/s | 233 kB 00:00 2026-02-25T16:35:15.749 INFO:teuthology.orchestra.run.trial129.stdout:(5/10): protobuf-3.14.0-17.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2026-02-25T16:35:15.772 INFO:teuthology.orchestra.run.trial129.stdout:(6/10): socat-1.7.4.1-8.el9.x86_64.rpm 12 MB/s | 303 kB 00:00 2026-02-25T16:35:15.794 INFO:teuthology.orchestra.run.trial129.stdout:(7/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.8 MB/s | 64 kB 00:00 2026-02-25T16:35:15.930 INFO:teuthology.orchestra.run.trial129.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 899 kB/s | 122 kB 00:00 2026-02-25T16:35:16.435 INFO:teuthology.orchestra.run.trial129.stdout:(9/10): ceph-test-20.3.0-5305.g0886691e.el9.x86 50 MB/s | 78 MB 00:01 2026-02-25T16:35:17.767 INFO:teuthology.orchestra.run.trial129.stdout:(10/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 742 kB/s | 2.1 MB 00:02 2026-02-25T16:35:17.771 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:17.771 INFO:teuthology.orchestra.run.trial129.stdout:Total 23 MB/s | 82 MB 00:03 2026-02-25T16:35:17.833 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:17.845 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:17.845 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:18.131 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:18.131 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:18.441 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:18.450 INFO:teuthology.orchestra.run.trial129.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-25T16:35:18.459 INFO:teuthology.orchestra.run.trial129.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-25T16:35:18.464 INFO:teuthology.orchestra.run.trial129.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-25T16:35:18.468 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-25T16:35:18.473 INFO:teuthology.orchestra.run.trial129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-25T16:35:18.475 INFO:teuthology.orchestra.run.trial129.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-25T16:35:18.485 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-25T16:35:18.493 INFO:teuthology.orchestra.run.trial129.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-25T16:35:18.494 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libcephsqlite-2:20.3.0-5305.g0886691e.el9.x86_64 9/10 2026-02-25T16:35:21.934 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5305.g0886691e.el9.x86_64 9/10 2026-02-25T16:35:21.953 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-test-2:20.3.0-5305.g0886691e.el9.x86_64 10/10 2026-02-25T16:35:22.307 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-test-2:20.3.0-5305.g0886691e.el9.x86_64 10/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-test-2:20.3.0-5305.g0886691e.el9.x86_64 1/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libcephsqlite-2:20.3.0-5305.g0886691e.el9.x86_64 2/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-25T16:35:22.308 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-25T16:35:22.309 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-25T16:35:22.369 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: ceph-test-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: libcephsqlite-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T16:35:22.370 INFO:teuthology.orchestra.run.trial129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T16:35:22.371 INFO:teuthology.orchestra.run.trial129.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T16:35:22.371 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:22.371 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:22.407 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph 2026-02-25T16:35:22.721 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:23 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:22.809 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:22.812 INFO:teuthology.orchestra.run.trial129.stdout:======================================================================================= 2026-02-25T16:35:22.812 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repository Size 2026-02-25T16:35:22.812 INFO:teuthology.orchestra.run.trial129.stdout:======================================================================================= 2026-02-25T16:35:22.812 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: ceph x86_64 2:20.3.0-5305.g0886691e.el9 ceph 6.5 k 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout:Installing dependencies: 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 57 k 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mds x86_64 2:20.3.0-5305.g0886691e.el9 ceph 2.4 M 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr x86_64 2:20.3.0-5305.g0886691e.el9 ceph 936 k 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 297 k 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mon x86_64 2:20.3.0-5305.g0886691e.el9 ceph 5.0 M 2026-02-25T16:35:22.813 INFO:teuthology.orchestra.run.trial129.stdout: ceph-osd x86_64 2:20.3.0-5305.g0886691e.el9 ceph 167 k 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: ceph-osd-classic x86_64 2:20.3.0-5305.g0886691e.el9 ceph 16 M 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: ceph-osd-crimson x86_64 2:20.3.0-5305.g0886691e.el9 ceph 19 M 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 17 k 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: cephadm noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 1.0 M 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T16:35:22.814 INFO:teuthology.orchestra.run.trial129.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T16:35:22.815 INFO:teuthology.orchestra.run.trial129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T16:35:22.816 INFO:teuthology.orchestra.run.trial129.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T16:35:22.817 INFO:teuthology.orchestra.run.trial129.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T16:35:22.818 INFO:teuthology.orchestra.run.trial129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T16:35:22.819 INFO:teuthology.orchestra.run.trial129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T16:35:22.820 INFO:teuthology.orchestra.run.trial129.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout:Installing weak dependencies: 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 184 k 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 51 M 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 7.4 M 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 22 k 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-rook noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 50 k 2026-02-25T16:35:22.821 INFO:teuthology.orchestra.run.trial129.stdout: ceph-volume noarch 2:20.3.0-5305.g0886691e.el9 ceph-noarch 302 k 2026-02-25T16:35:22.822 INFO:teuthology.orchestra.run.trial129.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T16:35:22.822 INFO:teuthology.orchestra.run.trial129.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T16:35:22.822 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:22.822 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:22.822 INFO:teuthology.orchestra.run.trial129.stdout:======================================================================================= 2026-02-25T16:35:22.822 INFO:teuthology.orchestra.run.trial129.stdout:Install 96 Packages 2026-02-25T16:35:22.822 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:22.823 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 158 M 2026-02-25T16:35:22.823 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 732 M 2026-02-25T16:35:22.823 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:23.451 INFO:teuthology.orchestra.run.trial129.stdout:(1/96): ceph-20.3.0-5305.g0886691e.el9.x86_64.r 104 kB/s | 6.5 kB 00:00 2026-02-25T16:35:23.536 INFO:teuthology.orchestra.run.trial129.stdout:(2/96): ceph-mgr-20.3.0-5305.g0886691e.el9.x86_ 6.2 MB/s | 936 kB 00:00 2026-02-25T16:35:23.563 INFO:teuthology.orchestra.run.trial129.stdout:(3/96): ceph-mds-20.3.0-5305.g0886691e.el9.x86_ 14 MB/s | 2.4 MB 00:00 2026-02-25T16:35:23.566 INFO:teuthology.orchestra.run.trial129.stdout:(4/96): ceph-osd-20.3.0-5305.g0886691e.el9.x86_ 5.4 MB/s | 167 kB 00:00 2026-02-25T16:35:23.743 INFO:teuthology.orchestra.run.trial129.stdout:(5/96): ceph-mon-20.3.0-5305.g0886691e.el9.x86_ 17 MB/s | 5.0 MB 00:00 2026-02-25T16:35:23.786 INFO:teuthology.orchestra.run.trial129.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5305.g08 1.3 MB/s | 57 kB 00:00 2026-02-25T16:35:23.835 INFO:teuthology.orchestra.run.trial129.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5305.g0886691e. 3.7 MB/s | 184 kB 00:00 2026-02-25T16:35:24.218 INFO:teuthology.orchestra.run.trial129.stdout:(8/96): ceph-osd-classic-20.3.0-5305.g0886691e. 24 MB/s | 16 MB 00:00 2026-02-25T16:35:24.555 INFO:teuthology.orchestra.run.trial129.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 22 MB/s | 7.4 MB 00:00 2026-02-25T16:35:24.580 INFO:teuthology.orchestra.run.trial129.stdout:(10/96): ceph-osd-crimson-20.3.0-5305.g0886691e 19 MB/s | 19 MB 00:01 2026-02-25T16:35:24.581 INFO:teuthology.orchestra.run.trial129.stdout:(11/96): ceph-mgr-k8sevents-20.3.0-5305.g088669 873 kB/s | 22 kB 00:00 2026-02-25T16:35:24.594 INFO:teuthology.orchestra.run.trial129.stdout:(12/96): ceph-mgr-rook-20.3.0-5305.g0886691e.el 3.7 MB/s | 50 kB 00:00 2026-02-25T16:35:24.598 INFO:teuthology.orchestra.run.trial129.stdout:(13/96): ceph-mgr-modules-core-20.3.0-5305.g088 17 MB/s | 297 kB 00:00 2026-02-25T16:35:24.613 INFO:teuthology.orchestra.run.trial129.stdout:(14/96): ceph-prometheus-alerts-20.3.0-5305.g08 915 kB/s | 17 kB 00:00 2026-02-25T16:35:24.622 INFO:teuthology.orchestra.run.trial129.stdout:(15/96): ceph-volume-20.3.0-5305.g0886691e.el9. 12 MB/s | 302 kB 00:00 2026-02-25T16:35:24.657 INFO:teuthology.orchestra.run.trial129.stdout:(16/96): cephadm-20.3.0-5305.g0886691e.el9.noar 24 MB/s | 1.0 MB 00:00 2026-02-25T16:35:24.719 INFO:teuthology.orchestra.run.trial129.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 648 kB/s | 40 kB 00:00 2026-02-25T16:35:24.731 INFO:teuthology.orchestra.run.trial129.stdout:(18/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.1 MB/s | 351 kB 00:00 2026-02-25T16:35:24.741 INFO:teuthology.orchestra.run.trial129.stdout:(19/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-25T16:35:24.776 INFO:teuthology.orchestra.run.trial129.stdout:(20/96): libgfortran-11.5.0-14.el9.x86_64.rpm 17 MB/s | 794 kB 00:00 2026-02-25T16:35:24.785 INFO:teuthology.orchestra.run.trial129.stdout:(21/96): libquadmath-11.5.0-14.el9.x86_64.rpm 4.2 MB/s | 184 kB 00:00 2026-02-25T16:35:24.803 INFO:teuthology.orchestra.run.trial129.stdout:(22/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 9.3 MB/s | 253 kB 00:00 2026-02-25T16:35:24.826 INFO:teuthology.orchestra.run.trial129.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.5 MB/s | 106 kB 00:00 2026-02-25T16:35:24.850 INFO:teuthology.orchestra.run.trial129.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.5 MB/s | 135 kB 00:00 2026-02-25T16:35:24.867 INFO:teuthology.orchestra.run.trial129.stdout:(25/96): python3-cryptography-36.0.1-5.el9.x86_ 15 MB/s | 1.2 MB 00:00 2026-02-25T16:35:24.874 INFO:teuthology.orchestra.run.trial129.stdout:(26/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-25T16:35:24.890 INFO:teuthology.orchestra.run.trial129.stdout:(27/96): python3-requests-2.25.1-10.el9.noarch. 5.2 MB/s | 126 kB 00:00 2026-02-25T16:35:24.900 INFO:teuthology.orchestra.run.trial129.stdout:(28/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.2 MB/s | 218 kB 00:00 2026-02-25T16:35:24.914 INFO:teuthology.orchestra.run.trial129.stdout:(29/96): unzip-6.0-59.el9.x86_64.rpm 7.6 MB/s | 182 kB 00:00 2026-02-25T16:35:24.924 INFO:teuthology.orchestra.run.trial129.stdout:(30/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-25T16:35:25.262 INFO:teuthology.orchestra.run.trial129.stdout:(31/96): flexiblas-3.0.4-9.el9.x86_64.rpm 85 kB/s | 30 kB 00:00 2026-02-25T16:35:25.504 INFO:teuthology.orchestra.run.trial129.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 61 kB/s | 15 kB 00:00 2026-02-25T16:35:25.527 INFO:teuthology.orchestra.run.trial129.stdout:(33/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 5.0 MB/s | 3.0 MB 00:00 2026-02-25T16:35:25.863 INFO:teuthology.orchestra.run.trial129.stdout:(34/96): ceph-mgr-dashboard-20.3.0-5305.g088669 25 MB/s | 51 MB 00:02 2026-02-25T16:35:25.863 INFO:teuthology.orchestra.run.trial129.stdout:(35/96): lua-5.4.4-4.el9.x86_64.rpm 560 kB/s | 188 kB 00:00 2026-02-25T16:35:25.893 INFO:teuthology.orchestra.run.trial129.stdout:(36/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 633 kB/s | 246 kB 00:00 2026-02-25T16:35:26.241 INFO:teuthology.orchestra.run.trial129.stdout:(37/96): openblas-0.3.29-1.el9.x86_64.rpm 111 kB/s | 42 kB 00:00 2026-02-25T16:35:26.364 INFO:teuthology.orchestra.run.trial129.stdout:(38/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 11 MB/s | 5.3 MB 00:00 2026-02-25T16:35:26.448 INFO:teuthology.orchestra.run.trial129.stdout:(39/96): python3-devel-3.9.25-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2026-02-25T16:35:26.552 INFO:teuthology.orchestra.run.trial129.stdout:(40/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 1.3 MB/s | 249 kB 00:00 2026-02-25T16:35:26.605 INFO:teuthology.orchestra.run.trial129.stdout:(41/96): python3-jmespath-1.0.1-1.el9.noarch.rp 303 kB/s | 48 kB 00:00 2026-02-25T16:35:26.710 INFO:teuthology.orchestra.run.trial129.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 1.1 MB/s | 177 kB 00:00 2026-02-25T16:35:26.815 INFO:teuthology.orchestra.run.trial129.stdout:(43/96): python3-babel-2.9.1-2.el9.noarch.rpm 6.5 MB/s | 6.0 MB 00:00 2026-02-25T16:35:26.852 INFO:teuthology.orchestra.run.trial129.stdout:(44/96): python3-markupsafe-1.1.1-12.el9.x86_64 245 kB/s | 35 kB 00:00 2026-02-25T16:35:27.001 INFO:teuthology.orchestra.run.trial129.stdout:(45/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 3.1 MB/s | 1.2 MB 00:00 2026-02-25T16:35:27.137 INFO:teuthology.orchestra.run.trial129.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 567 kB/s | 77 kB 00:00 2026-02-25T16:35:27.139 INFO:teuthology.orchestra.run.trial129.stdout:(47/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 1.5 MB/s | 442 kB 00:00 2026-02-25T16:35:27.448 INFO:teuthology.orchestra.run.trial129.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 509 kB/s | 157 kB 00:00 2026-02-25T16:35:27.688 INFO:teuthology.orchestra.run.trial129.stdout:(49/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 7.0 MB/s | 6.1 MB 00:00 2026-02-25T16:35:27.807 INFO:teuthology.orchestra.run.trial129.stdout:(50/96): python3-protobuf-3.14.0-17.el9.noarch. 399 kB/s | 267 kB 00:00 2026-02-25T16:35:28.134 INFO:teuthology.orchestra.run.trial129.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 120 kB/s | 54 kB 00:00 2026-02-25T16:35:28.255 INFO:teuthology.orchestra.run.trial129.stdout:(52/96): python3-pyasn1-modules-0.4.8-7.el9.noa 344 kB/s | 277 kB 00:00 2026-02-25T16:35:28.355 INFO:teuthology.orchestra.run.trial129.stdout:(53/96): python3-toml-0.10.2-6.el9.noarch.rpm 188 kB/s | 42 kB 00:00 2026-02-25T16:35:28.545 INFO:teuthology.orchestra.run.trial129.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 651 kB/s | 189 kB 00:00 2026-02-25T16:35:28.614 INFO:teuthology.orchestra.run.trial129.stdout:(55/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 350 kB/s | 90 kB 00:00 2026-02-25T16:35:29.104 INFO:teuthology.orchestra.run.trial129.stdout:(56/96): lua-devel-5.4.4-4.el9.x86_64.rpm 40 kB/s | 22 kB 00:00 2026-02-25T16:35:29.381 INFO:teuthology.orchestra.run.trial129.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.9 MB/s | 551 kB 00:00 2026-02-25T16:35:29.417 INFO:teuthology.orchestra.run.trial129.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 545 kB/s | 19 kB 00:00 2026-02-25T16:35:29.495 INFO:teuthology.orchestra.run.trial129.stdout:(59/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 11 MB/s | 19 MB 00:01 2026-02-25T16:35:29.495 INFO:teuthology.orchestra.run.trial129.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 1.9 MB/s | 151 kB 00:00 2026-02-25T16:35:29.532 INFO:teuthology.orchestra.run.trial129.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 804 kB/s | 29 kB 00:00 2026-02-25T16:35:29.570 INFO:teuthology.orchestra.run.trial129.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.6 MB/s | 60 kB 00:00 2026-02-25T16:35:29.607 INFO:teuthology.orchestra.run.trial129.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.1 MB/s | 43 kB 00:00 2026-02-25T16:35:29.643 INFO:teuthology.orchestra.run.trial129.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 895 kB/s | 32 kB 00:00 2026-02-25T16:35:29.679 INFO:teuthology.orchestra.run.trial129.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 391 kB/s | 14 kB 00:00 2026-02-25T16:35:29.722 INFO:teuthology.orchestra.run.trial129.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.0 MB/s | 173 kB 00:00 2026-02-25T16:35:29.758 INFO:teuthology.orchestra.run.trial129.stdout:(67/96): python3-asyncssh-2.13.2-5.el9.noarch.r 2.0 MB/s | 548 kB 00:00 2026-02-25T16:35:29.772 INFO:teuthology.orchestra.run.trial129.stdout:(68/96): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2026-02-25T16:35:29.804 INFO:teuthology.orchestra.run.trial129.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 5.4 MB/s | 254 kB 00:00 2026-02-25T16:35:29.867 INFO:teuthology.orchestra.run.trial129.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 2.2 MB/s | 144 kB 00:00 2026-02-25T16:35:29.885 INFO:teuthology.orchestra.run.trial129.stdout:(71/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 18 MB/s | 2.0 MB 00:00 2026-02-25T16:35:29.908 INFO:teuthology.orchestra.run.trial129.stdout:(72/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.3 MB/s | 139 kB 00:00 2026-02-25T16:35:29.922 INFO:teuthology.orchestra.run.trial129.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.5 MB/s | 56 kB 00:00 2026-02-25T16:35:29.944 INFO:teuthology.orchestra.run.trial129.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 298 kB/s | 11 kB 00:00 2026-02-25T16:35:29.958 INFO:teuthology.orchestra.run.trial129.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 489 kB/s | 18 kB 00:00 2026-02-25T16:35:29.980 INFO:teuthology.orchestra.run.trial129.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 641 kB/s | 23 kB 00:00 2026-02-25T16:35:29.994 INFO:teuthology.orchestra.run.trial129.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 544 kB/s | 20 kB 00:00 2026-02-25T16:35:30.016 INFO:teuthology.orchestra.run.trial129.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 542 kB/s | 19 kB 00:00 2026-02-25T16:35:30.031 INFO:teuthology.orchestra.run.trial129.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 731 kB/s | 26 kB 00:00 2026-02-25T16:35:30.068 INFO:teuthology.orchestra.run.trial129.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.1 MB/s | 79 kB 00:00 2026-02-25T16:35:30.099 INFO:teuthology.orchestra.run.trial129.stdout:(81/96): python3-kubernetes-26.1.0-3.el9.noarch 12 MB/s | 1.0 MB 00:00 2026-02-25T16:35:30.102 INFO:teuthology.orchestra.run.trial129.stdout:(82/96): protobuf-compiler-3.14.0-17.el9.x86_64 579 kB/s | 862 kB 00:01 2026-02-25T16:35:30.105 INFO:teuthology.orchestra.run.trial129.stdout:(83/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.3 MB/s | 86 kB 00:00 2026-02-25T16:35:30.136 INFO:teuthology.orchestra.run.trial129.stdout:(84/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.5 MB/s | 58 kB 00:00 2026-02-25T16:35:30.142 INFO:teuthology.orchestra.run.trial129.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.4 MB/s | 90 kB 00:00 2026-02-25T16:35:30.172 INFO:teuthology.orchestra.run.trial129.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 843 kB/s | 31 kB 00:00 2026-02-25T16:35:30.180 INFO:teuthology.orchestra.run.trial129.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 4.8 MB/s | 188 kB 00:00 2026-02-25T16:35:30.206 INFO:teuthology.orchestra.run.trial129.stdout:(88/96): python3-portend-3.1.0-2.el9.noarch.rpm 158 kB/s | 16 kB 00:00 2026-02-25T16:35:30.209 INFO:teuthology.orchestra.run.trial129.stdout:(89/96): python3-rsa-4.9-2.el9.noarch.rpm 1.6 MB/s | 59 kB 00:00 2026-02-25T16:35:30.219 INFO:teuthology.orchestra.run.trial129.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.2 MB/s | 125 kB 00:00 2026-02-25T16:35:30.245 INFO:teuthology.orchestra.run.trial129.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 2.3 MB/s | 86 kB 00:00 2026-02-25T16:35:30.256 INFO:teuthology.orchestra.run.trial129.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.4 MB/s | 90 kB 00:00 2026-02-25T16:35:30.277 INFO:teuthology.orchestra.run.trial129.stdout:(93/96): python3-tempora-5.0.0-2.el9.noarch.rpm 506 kB/s | 36 kB 00:00 2026-02-25T16:35:30.282 INFO:teuthology.orchestra.run.trial129.stdout:(94/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.3 MB/s | 48 kB 00:00 2026-02-25T16:35:30.308 INFO:teuthology.orchestra.run.trial129.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 422 kB/s | 22 kB 00:00 2026-02-25T16:35:30.319 INFO:teuthology.orchestra.run.trial129.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 467 kB/s | 20 kB 00:00 2026-02-25T16:35:30.321 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:30.321 INFO:teuthology.orchestra.run.trial129.stdout:Total 21 MB/s | 158 MB 00:07 2026-02-25T16:35:30.738 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:30.775 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:30.775 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:31.377 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:31.378 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:32.008 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:32.010 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T16:35:32.014 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T16:35:32.036 INFO:teuthology.orchestra.run.trial129.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T16:35:32.070 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T16:35:32.077 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T16:35:32.079 INFO:teuthology.orchestra.run.trial129.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T16:35:32.084 INFO:teuthology.orchestra.run.trial129.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T16:35:32.107 INFO:teuthology.orchestra.run.trial129.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T16:35:32.120 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T16:35:32.123 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T16:35:32.129 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T16:35:32.131 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T16:35:32.142 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T16:35:32.148 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T16:35:32.173 INFO:teuthology.orchestra.run.trial129.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T16:35:32.176 INFO:teuthology.orchestra.run.trial129.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T16:35:32.226 INFO:teuthology.orchestra.run.trial129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T16:35:32.239 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T16:35:32.242 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T16:35:32.246 INFO:teuthology.orchestra.run.trial129.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T16:35:32.252 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T16:35:32.256 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T16:35:32.260 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T16:35:32.274 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T16:35:32.278 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T16:35:32.283 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T16:35:32.295 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T16:35:32.304 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T16:35:32.310 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T16:35:32.329 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T16:35:32.340 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T16:35:32.356 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T16:35:32.360 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T16:35:32.368 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T16:35:32.408 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T16:35:32.730 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T16:35:32.740 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T16:35:32.744 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T16:35:32.749 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T16:35:32.751 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T16:35:32.780 INFO:teuthology.orchestra.run.trial129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T16:35:32.823 INFO:teuthology.orchestra.run.trial129.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T16:35:32.831 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T16:35:32.837 INFO:teuthology.orchestra.run.trial129.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T16:35:32.840 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T16:35:32.846 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T16:35:32.849 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T16:35:32.856 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T16:35:32.860 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T16:35:32.888 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T16:35:32.897 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T16:35:32.903 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T16:35:32.910 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T16:35:32.944 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T16:35:33.166 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T16:35:33.190 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T16:35:33.194 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T16:35:33.253 INFO:teuthology.orchestra.run.trial129.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T16:35:33.255 INFO:teuthology.orchestra.run.trial129.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T16:35:33.276 INFO:teuthology.orchestra.run.trial129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T16:35:33.603 INFO:teuthology.orchestra.run.trial129.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T16:35:33.676 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T16:35:34.349 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T16:35:34.365 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T16:35:34.375 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T16:35:34.382 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T16:35:34.395 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T16:35:34.412 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T16:35:34.487 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T16:35:34.496 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T16:35:34.520 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T16:35:34.549 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T16:35:34.600 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T16:35:34.606 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T16:35:34.610 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T16:35:34.611 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T16:35:34.629 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T16:35:34.629 INFO:teuthology.orchestra.run.trial129.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T16:35:34.629 INFO:teuthology.orchestra.run.trial129.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T16:35:34.629 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:34.637 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T16:35:34.663 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T16:35:34.663 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T16:35:34.663 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:34.700 INFO:teuthology.orchestra.run.trial129.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T16:35:35.317 INFO:teuthology.orchestra.run.trial129.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T16:35:36.074 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-osd-classic-2:20.3.0-5305.g0886691e.el9.x86_6 80/96 2026-02-25T16:35:36.137 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-osd-crimson-2:20.3.0-5305.g0886691e.el9.x86_6 81/96 2026-02-25T16:35:36.140 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-volume-2:20.3.0-5305.g0886691e.el9.noarch 82/96 2026-02-25T16:35:36.154 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-volume-2:20.3.0-5305.g0886691e.el9.noarch 82/96 2026-02-25T16:35:36.154 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:36.154 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T16:35:36.154 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:36.165 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-osd-2:20.3.0-5305.g0886691e.el9.x86_64 83/96 2026-02-25T16:35:36.189 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-osd-2:20.3.0-5305.g0886691e.el9.x86_64 83/96 2026-02-25T16:35:36.189 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:36.189 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T16:35:36.189 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T16:35:36.189 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T16:35:36.189 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:36.249 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: cephadm-2:20.3.0-5305.g0886691e.el9.noarch 84/96 2026-02-25T16:35:36.251 INFO:teuthology.orchestra.run.trial129.stdout: Installing : cephadm-2:20.3.0-5305.g0886691e.el9.noarch 84/96 2026-02-25T16:35:36.260 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5305.g0886691e.el9 85/96 2026-02-25T16:35:36.284 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5305.g0886691e.el 86/96 2026-02-25T16:35:36.285 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5305.g0886691e.el9.noarc 87/96 2026-02-25T16:35:39.166 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5305.g0886691e.el9.noarc 87/96 2026-02-25T16:35:39.172 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5305.g0886691e.el9.noa 88/96 2026-02-25T16:35:39.628 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5305.g0886691e.el9.noa 88/96 2026-02-25T16:35:39.630 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5305.g08866 89/96 2026-02-25T16:35:39.643 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5305.g08866 89/96 2026-02-25T16:35:39.644 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5305.g0886691e.el9.noa 90/96 2026-02-25T16:35:39.706 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5305.g0886691e.el9.noa 90/96 2026-02-25T16:35:39.748 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5305.g0886691e.el9. 91/96 2026-02-25T16:35:39.749 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mgr-2:20.3.0-5305.g0886691e.el9.x86_64 92/96 2026-02-25T16:35:39.772 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5305.g0886691e.el9.x86_64 92/96 2026-02-25T16:35:39.772 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:39.772 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T16:35:39.772 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T16:35:39.772 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T16:35:39.773 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:39.785 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mgr-rook-2:20.3.0-5305.g0886691e.el9.noarch 93/96 2026-02-25T16:35:40.007 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5305.g0886691e.el9.noarch 93/96 2026-02-25T16:35:40.008 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mon-2:20.3.0-5305.g0886691e.el9.x86_64 94/96 2026-02-25T16:35:40.031 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mon-2:20.3.0-5305.g0886691e.el9.x86_64 94/96 2026-02-25T16:35:40.032 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:40.032 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T16:35:40.032 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T16:35:40.032 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T16:35:40.032 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:40.126 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-mds-2:20.3.0-5305.g0886691e.el9.x86_64 95/96 2026-02-25T16:35:40.147 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-mds-2:20.3.0-5305.g0886691e.el9.x86_64 95/96 2026-02-25T16:35:40.147 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:40.147 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T16:35:40.147 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T16:35:40.147 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T16:35:40.147 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:40.148 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-2:20.3.0-5305.g0886691e.el9.x86_64 96/96 2026-02-25T16:35:40.155 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5305.g0886691e.el9.x86_6 96/96 2026-02-25T16:35:40.180 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5305.g0886691e.el9.x86_6 96/96 2026-02-25T16:35:41.004 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-2:20.3.0-5305.g0886691e.el9.x86_64 96/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-2:20.3.0-5305.g0886691e.el9.x86_64 1/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mds-2:20.3.0-5305.g0886691e.el9.x86_64 2/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mgr-2:20.3.0-5305.g0886691e.el9.x86_64 3/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mon-2:20.3.0-5305.g0886691e.el9.x86_64 4/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-osd-2:20.3.0-5305.g0886691e.el9.x86_64 5/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-osd-classic-2:20.3.0-5305.g0886691e.el9.x86_6 6/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5305.g0886691e.el9.x86_6 7/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5305.g0886691e.el 8/96 2026-02-25T16:35:41.005 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5305.g0886691e.el9.noarc 9/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5305.g0886691e.el9.noa 10/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5305.g08866 11/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5305.g0886691e.el9.noa 12/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5305.g0886691e.el9. 13/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5305.g0886691e.el9.noarch 14/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5305.g0886691e.el9 15/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-volume-2:20.3.0-5305.g0886691e.el9.noarch 16/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : cephadm-2:20.3.0-5305.g0886691e.el9.noarch 17/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T16:35:41.006 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T16:35:41.007 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T16:35:41.008 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T16:35:41.009 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T16:35:41.010 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T16:35:41.011 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T16:35:41.012 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T16:35:41.013 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T16:35:41.072 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T16:35:41.072 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:41.072 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:41.072 INFO:teuthology.orchestra.run.trial129.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T16:35:41.072 INFO:teuthology.orchestra.run.trial129.stdout: ceph-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:41.072 INFO:teuthology.orchestra.run.trial129.stdout: ceph-grafana-dashboards-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.072 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mds-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-cephadm-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-dashboard-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-k8sevents-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-modules-core-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mgr-rook-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-mon-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-osd-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-osd-classic-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:41.073 INFO:teuthology.orchestra.run.trial129.stdout: ceph-osd-crimson-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: ceph-prometheus-alerts-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: ceph-volume-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: cephadm-2:20.3.0-5305.g0886691e.el9.noarch 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T16:35:41.074 INFO:teuthology.orchestra.run.trial129.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T16:35:41.075 INFO:teuthology.orchestra.run.trial129.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T16:35:41.076 INFO:teuthology.orchestra.run.trial129.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T16:35:41.077 INFO:teuthology.orchestra.run.trial129.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T16:35:41.078 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T16:35:41.079 INFO:teuthology.orchestra.run.trial129.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T16:35:41.080 INFO:teuthology.orchestra.run.trial129.stdout: zip-3.0-35.el9.x86_64 2026-02-25T16:35:41.081 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:41.081 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:41.134 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-base 2026-02-25T16:35:41.453 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:42 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:41.519 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-base-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:41.537 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:41.538 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:41.538 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:41.560 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install cephadm 2026-02-25T16:35:41.877 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:42 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:41.943 INFO:teuthology.orchestra.run.trial129.stdout:Package cephadm-2:20.3.0-5305.g0886691e.el9.noarch is already installed. 2026-02-25T16:35:41.961 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:41.962 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:41.962 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:41.984 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T16:35:42.304 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:43 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:42.388 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:42.388 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repo Size 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5305.g0886691e.el9 ceph 147 k 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout:Install 1 Package 2026-02-25T16:35:42.389 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:42.390 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 147 k 2026-02-25T16:35:42.390 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 419 k 2026-02-25T16:35:42.390 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:42.512 INFO:teuthology.orchestra.run.trial129.stdout:ceph-immutable-object-cache-20.3.0-5305.g088669 1.2 MB/s | 147 kB 00:00 2026-02-25T16:35:42.513 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:42.513 INFO:teuthology.orchestra.run.trial129.stdout:Total 1.2 MB/s | 147 kB 00:00 2026-02-25T16:35:42.513 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:42.517 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:42.517 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:42.548 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:42.548 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:42.603 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:42.605 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5305.g0886691e. 1/1 2026-02-25T16:35:42.628 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5305.g0886691e. 1/1 2026-02-25T16:35:42.628 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:42.628 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T16:35:42.628 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:42.918 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5305.g0886691e. 1/1 2026-02-25T16:35:42.918 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:42.918 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:42.918 INFO:teuthology.orchestra.run.trial129.stdout: ceph-immutable-object-cache-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:42.918 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:42.918 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:42.947 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-mgr 2026-02-25T16:35:43.265 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:44 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:43.332 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-mgr-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:43.351 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:43.352 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:43.352 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:43.373 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T16:35:43.692 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:44 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:43.757 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-mgr-dashboard-2:20.3.0-5305.g0886691e.el9.noarch is already installed. 2026-02-25T16:35:43.776 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:43.777 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:43.777 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:43.798 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T16:35:44.116 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:45 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:44.182 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5305.g0886691e.el9.noarch is already installed. 2026-02-25T16:35:44.201 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:44.201 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:44.202 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:44.223 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-mgr-rook 2026-02-25T16:35:44.547 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:45 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:44.611 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-mgr-rook-2:20.3.0-5305.g0886691e.el9.noarch is already installed. 2026-02-25T16:35:44.630 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:44.631 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:44.631 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:44.652 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T16:35:44.967 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:45 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:45.032 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-mgr-cephadm-2:20.3.0-5305.g0886691e.el9.noarch is already installed. 2026-02-25T16:35:45.051 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:45.051 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:45.052 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:45.073 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-osd 2026-02-25T16:35:45.392 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:46 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:45.458 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-osd-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:45.476 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:45.477 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:45.477 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:45.499 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-osd-classic 2026-02-25T16:35:45.819 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:46 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:45.884 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-osd-classic-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:45.903 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:45.903 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:45.903 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:45.925 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-fuse 2026-02-25T16:35:46.244 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:47 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repo Size 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout: ceph-fuse x86_64 2:20.3.0-5305.g0886691e.el9 ceph 987 k 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:Install 1 Package 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 987 k 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 2.9 M 2026-02-25T16:35:46.328 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:46.464 INFO:teuthology.orchestra.run.trial129.stdout:ceph-fuse-20.3.0-5305.g0886691e.el9.x86_64.rpm 7.1 MB/s | 987 kB 00:00 2026-02-25T16:35:46.465 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:46.465 INFO:teuthology.orchestra.run.trial129.stdout:Total 7.1 MB/s | 987 kB 00:00 2026-02-25T16:35:46.465 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:46.469 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:46.469 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:46.503 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:46.503 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:46.595 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:46.613 INFO:teuthology.orchestra.run.trial129.stdout: Installing : ceph-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:46.854 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:46.916 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : ceph-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:46.916 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:46.916 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:46.916 INFO:teuthology.orchestra.run.trial129.stdout: ceph-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:46.916 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:46.916 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:46.944 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install ceph-volume 2026-02-25T16:35:47.262 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:48 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:47.331 INFO:teuthology.orchestra.run.trial129.stdout:Package ceph-volume-2:20.3.0-5305.g0886691e.el9.noarch is already installed. 2026-02-25T16:35:47.349 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:47.350 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:47.351 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:47.372 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install librados-devel 2026-02-25T16:35:47.686 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:48 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repo Size 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout: librados-devel x86_64 2:20.3.0-5305.g0886691e.el9 ceph 123 k 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout:Install 1 Package 2026-02-25T16:35:47.770 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:47.771 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 123 k 2026-02-25T16:35:47.771 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 437 k 2026-02-25T16:35:47.771 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:47.869 INFO:teuthology.orchestra.run.trial129.stdout:librados-devel-20.3.0-5305.g0886691e.el9.x86_64 1.2 MB/s | 123 kB 00:00 2026-02-25T16:35:47.869 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:47.869 INFO:teuthology.orchestra.run.trial129.stdout:Total 1.2 MB/s | 123 kB 00:00 2026-02-25T16:35:47.869 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:47.873 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:47.873 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:47.905 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:47.906 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:47.964 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:47.981 INFO:teuthology.orchestra.run.trial129.stdout: Installing : librados-devel-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:48.066 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: librados-devel-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:48.129 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : librados-devel-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:48.129 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:48.129 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:48.130 INFO:teuthology.orchestra.run.trial129.stdout: librados-devel-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:48.130 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:48.130 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:48.156 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install libcephfs2 2026-02-25T16:35:48.478 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:49 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:48.545 INFO:teuthology.orchestra.run.trial129.stdout:Package libcephfs2-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:48.564 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:48.565 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:48.565 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:48.590 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install libcephfs-devel 2026-02-25T16:35:48.911 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:49 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:48.995 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repo Size 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs-devel x86_64 2:20.3.0-5305.g0886691e.el9 ceph 37 k 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:48.996 INFO:teuthology.orchestra.run.trial129.stdout:Install 1 Package 2026-02-25T16:35:48.997 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:48.997 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 37 k 2026-02-25T16:35:48.997 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 171 k 2026-02-25T16:35:48.997 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:49.123 INFO:teuthology.orchestra.run.trial129.stdout:libcephfs-devel-20.3.0-5305.g0886691e.el9.x86_6 294 kB/s | 37 kB 00:00 2026-02-25T16:35:49.124 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:49.124 INFO:teuthology.orchestra.run.trial129.stdout:Total 292 kB/s | 37 kB 00:00 2026-02-25T16:35:49.124 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:49.126 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:49.126 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:49.128 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:49.129 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:49.151 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:49.168 INFO:teuthology.orchestra.run.trial129.stdout: Installing : libcephfs-devel-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:49.231 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:49.293 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : libcephfs-devel-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:49.293 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:49.293 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:49.293 INFO:teuthology.orchestra.run.trial129.stdout: libcephfs-devel-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:49.294 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:49.294 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:49.321 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install librados2 2026-02-25T16:35:49.641 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:50 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:49.706 INFO:teuthology.orchestra.run.trial129.stdout:Package librados2-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:49.725 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:49.726 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:49.726 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:49.747 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install librbd1 2026-02-25T16:35:50.063 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:51 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:50.129 INFO:teuthology.orchestra.run.trial129.stdout:Package librbd1-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:50.148 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:50.148 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:50.148 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:50.169 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install python3-rados 2026-02-25T16:35:50.489 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:51 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:50.554 INFO:teuthology.orchestra.run.trial129.stdout:Package python3-rados-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:50.573 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:50.573 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:50.573 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:50.595 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install python3-rgw 2026-02-25T16:35:50.913 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:51 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:50.977 INFO:teuthology.orchestra.run.trial129.stdout:Package python3-rgw-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:50.995 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:50.996 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:50.996 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:51.017 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install python3-cephfs 2026-02-25T16:35:51.330 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:52 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:51.395 INFO:teuthology.orchestra.run.trial129.stdout:Package python3-cephfs-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:51.414 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:51.415 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:51.415 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:51.436 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install python3-rbd 2026-02-25T16:35:51.753 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:52 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:51.817 INFO:teuthology.orchestra.run.trial129.stdout:Package python3-rbd-2:20.3.0-5305.g0886691e.el9.x86_64 is already installed. 2026-02-25T16:35:51.836 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:51.836 INFO:teuthology.orchestra.run.trial129.stdout:Nothing to do. 2026-02-25T16:35:51.836 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:51.857 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install rbd-fuse 2026-02-25T16:35:52.180 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:53 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:52.264 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:52.264 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:52.264 INFO:teuthology.orchestra.run.trial129.stdout: Package Architecture Version Repository Size 2026-02-25T16:35:52.264 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:52.264 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout: rbd-fuse x86_64 2:20.3.0-5305.g0886691e.el9 ceph 84 k 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout:Install 1 Package 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 84 k 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 209 k 2026-02-25T16:35:52.265 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:52.359 INFO:teuthology.orchestra.run.trial129.stdout:rbd-fuse-20.3.0-5305.g0886691e.el9.x86_64.rpm 905 kB/s | 84 kB 00:00 2026-02-25T16:35:52.359 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:52.360 INFO:teuthology.orchestra.run.trial129.stdout:Total 895 kB/s | 84 kB 00:00 2026-02-25T16:35:52.360 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:52.364 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:52.364 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:52.394 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:52.395 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:52.446 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:52.464 INFO:teuthology.orchestra.run.trial129.stdout: Installing : rbd-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:52.552 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:52.616 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : rbd-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:52.616 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:52.616 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:52.616 INFO:teuthology.orchestra.run.trial129.stdout: rbd-fuse-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:52.616 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:52.616 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:52.644 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install rbd-mirror 2026-02-25T16:35:52.965 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:53 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:53.047 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:53.048 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:53.048 INFO:teuthology.orchestra.run.trial129.stdout: Package Arch Version Repo Size 2026-02-25T16:35:53.048 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:53.048 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:53.048 INFO:teuthology.orchestra.run.trial129.stdout: rbd-mirror x86_64 2:20.3.0-5305.g0886691e.el9 ceph 2.9 M 2026-02-25T16:35:53.048 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:53.049 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:53.049 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:53.049 INFO:teuthology.orchestra.run.trial129.stdout:Install 1 Package 2026-02-25T16:35:53.049 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:53.049 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 2.9 M 2026-02-25T16:35:53.049 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 11 M 2026-02-25T16:35:53.049 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:53.209 INFO:teuthology.orchestra.run.trial129.stdout:rbd-mirror-20.3.0-5305.g0886691e.el9.x86_64.rpm 18 MB/s | 2.9 MB 00:00 2026-02-25T16:35:53.209 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:53.209 INFO:teuthology.orchestra.run.trial129.stdout:Total 18 MB/s | 2.9 MB 00:00 2026-02-25T16:35:53.209 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:53.215 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:53.215 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:53.254 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:53.254 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:53.431 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:53.432 INFO:teuthology.orchestra.run.trial129.stdout: Installing : rbd-mirror-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:53.456 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:53.456 INFO:teuthology.orchestra.run.trial129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T16:35:53.456 INFO:teuthology.orchestra.run.trial129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T16:35:53.456 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T16:35:53.456 INFO:teuthology.orchestra.run.trial129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T16:35:53.456 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:53.757 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : rbd-mirror-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:53.757 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:53.757 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:53.757 INFO:teuthology.orchestra.run.trial129.stdout: rbd-mirror-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:53.757 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:53.757 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:53.784 DEBUG:teuthology.orchestra.run.trial129:> sudo yum -y install rbd-nbd 2026-02-25T16:35:54.107 INFO:teuthology.orchestra.run.trial129.stdout:Last metadata expiration check: 0:00:55 ago on Wed 25 Feb 2026 04:34:59 PM UTC. 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout:Dependencies resolved. 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout: Package Architecture Version Repository Size 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout:Installing: 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout: rbd-nbd x86_64 2:20.3.0-5305.g0886691e.el9 ceph 172 k 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:54.192 INFO:teuthology.orchestra.run.trial129.stdout:Transaction Summary 2026-02-25T16:35:54.193 INFO:teuthology.orchestra.run.trial129.stdout:================================================================================ 2026-02-25T16:35:54.193 INFO:teuthology.orchestra.run.trial129.stdout:Install 1 Package 2026-02-25T16:35:54.193 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:54.193 INFO:teuthology.orchestra.run.trial129.stdout:Total download size: 172 k 2026-02-25T16:35:54.193 INFO:teuthology.orchestra.run.trial129.stdout:Installed size: 462 k 2026-02-25T16:35:54.193 INFO:teuthology.orchestra.run.trial129.stdout:Downloading Packages: 2026-02-25T16:35:54.322 INFO:teuthology.orchestra.run.trial129.stdout:rbd-nbd-20.3.0-5305.g0886691e.el9.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2026-02-25T16:35:54.322 INFO:teuthology.orchestra.run.trial129.stdout:-------------------------------------------------------------------------------- 2026-02-25T16:35:54.322 INFO:teuthology.orchestra.run.trial129.stdout:Total 1.3 MB/s | 172 kB 00:00 2026-02-25T16:35:54.322 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction check 2026-02-25T16:35:54.327 INFO:teuthology.orchestra.run.trial129.stdout:Transaction check succeeded. 2026-02-25T16:35:54.327 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction test 2026-02-25T16:35:54.358 INFO:teuthology.orchestra.run.trial129.stdout:Transaction test succeeded. 2026-02-25T16:35:54.358 INFO:teuthology.orchestra.run.trial129.stdout:Running transaction 2026-02-25T16:35:54.414 INFO:teuthology.orchestra.run.trial129.stdout: Preparing : 1/1 2026-02-25T16:35:54.432 INFO:teuthology.orchestra.run.trial129.stdout: Installing : rbd-nbd-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:54.520 INFO:teuthology.orchestra.run.trial129.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:54.584 INFO:teuthology.orchestra.run.trial129.stdout: Verifying : rbd-nbd-2:20.3.0-5305.g0886691e.el9.x86_64 1/1 2026-02-25T16:35:54.585 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:54.585 INFO:teuthology.orchestra.run.trial129.stdout:Installed: 2026-02-25T16:35:54.585 INFO:teuthology.orchestra.run.trial129.stdout: rbd-nbd-2:20.3.0-5305.g0886691e.el9.x86_64 2026-02-25T16:35:54.585 INFO:teuthology.orchestra.run.trial129.stdout: 2026-02-25T16:35:54.585 INFO:teuthology.orchestra.run.trial129.stdout:Complete! 2026-02-25T16:35:54.616 DEBUG:teuthology.parallel:result is None 2026-02-25T16:35:54.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T16:35:54.791 DEBUG:teuthology.orchestra.run.trial129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T16:35:54.814 INFO:teuthology.orchestra.run.trial129.stdout:20.3.0-5305.g0886691e.el9 2026-02-25T16:35:54.815 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5305.g0886691e.el9 2026-02-25T16:35:54.815 INFO:teuthology.task.install:The correct ceph version 20.3.0-5305.g0886691e is installed. 2026-02-25T16:35:54.817 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T16:35:54.817 DEBUG:teuthology.orchestra.run.trial129:> set -ex 2026-02-25T16:35:54.817 DEBUG:teuthology.orchestra.run.trial129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T16:35:54.879 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T16:35:54.879 DEBUG:teuthology.orchestra.run.trial129:> set -ex 2026-02-25T16:35:54.879 DEBUG:teuthology.orchestra.run.trial129:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T16:35:54.943 DEBUG:teuthology.orchestra.run.trial129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T16:35:55.007 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T16:35:55.008 DEBUG:teuthology.orchestra.run.trial129:> set -ex 2026-02-25T16:35:55.008 DEBUG:teuthology.orchestra.run.trial129:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T16:35:55.071 DEBUG:teuthology.orchestra.run.trial129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T16:35:55.135 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T16:35:55.136 DEBUG:teuthology.orchestra.run.trial129:> set -ex 2026-02-25T16:35:55.136 DEBUG:teuthology.orchestra.run.trial129:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T16:35:55.199 DEBUG:teuthology.orchestra.run.trial129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T16:35:55.272 INFO:teuthology.run_tasks:Running task workunit... 2026-02-25T16:35:55.369 INFO:tasks.workunit:Pulling workunits from ref 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T16:35:55.369 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-25T16:35:55.369 INFO:tasks.workunit:timeout=3h 2026-02-25T16:35:55.369 INFO:tasks.workunit:cleanup=True 2026-02-25T16:35:55.370 DEBUG:teuthology.orchestra.run.trial129:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-25T16:35:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T16:35:55.392 INFO:teuthology.orchestra.run.trial129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-25T16:35:55.392 DEBUG:teuthology.orchestra.run.trial129:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T16:35:55.455 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-25T16:35:55.455 DEBUG:teuthology.orchestra.run.trial129:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-25T16:35:55.514 DEBUG:teuthology.orchestra.run.trial129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T16:35:55.576 INFO:tasks.workunit.client.0.trial129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-25T16:36:22.063 INFO:tasks.workunit.client.0.trial129.stderr:Note: switching to '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'. 2026-02-25T16:36:22.063 INFO:tasks.workunit.client.0.trial129.stderr: 2026-02-25T16:36:22.063 INFO:tasks.workunit.client.0.trial129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-25T16:36:22.063 INFO:tasks.workunit.client.0.trial129.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-25T16:36:22.063 INFO:tasks.workunit.client.0.trial129.stderr:state without impacting any branches by switching back to a branch. 2026-02-25T16:36:22.063 INFO:tasks.workunit.client.0.trial129.stderr: 2026-02-25T16:36:22.063 INFO:tasks.workunit.client.0.trial129.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr: 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr: git switch -c 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr: 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr:Or undo this operation with: 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr: 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr: git switch - 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr: 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr: 2026-02-25T16:36:22.064 INFO:tasks.workunit.client.0.trial129.stderr:HEAD is now at 0886691e8f7 doc/rados/operations/health-checks: Added MON_COLOCATED health warning Update the Health Checks doc page with MON_COLOCATED warning 2026-02-25T16:36:22.071 DEBUG:teuthology.orchestra.run.trial129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-25T16:36:22.128 DEBUG:teuthology.orchestra.run.trial129:> set -ex 2026-02-25T16:36:22.129 DEBUG:teuthology.orchestra.run.trial129:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-25T16:36:22.187 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2026-02-25T16:36:22.188 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2026-02-25T16:36:22.188 DEBUG:teuthology.orchestra.run.trial129:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2026-02-25T16:36:22.249 INFO:tasks.workunit.client.0.trial129.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2026-02-25T16:36:22.249 INFO:tasks.workunit.client.0.trial129.stderr:++ TIMEOUT=300 2026-02-25T16:36:22.249 INFO:tasks.workunit.client.0.trial129.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-25T16:36:22.249 INFO:tasks.workunit.client.0.trial129.stderr:++ MAX_TIMEOUT=15 2026-02-25T16:36:22.249 INFO:tasks.workunit.client.0.trial129.stderr:++ PG_NUM=4 2026-02-25T16:36:22.249 INFO:tasks.workunit.client.0.trial129.stderr:++ TMPDIR=/tmp 2026-02-25T16:36:22.249 INFO:tasks.workunit.client.0.trial129.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-25T16:36:22.250 INFO:tasks.workunit.client.0.trial129.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-25T16:36:22.250 INFO:tasks.workunit.client.0.trial129.stderr:+++ uname 2026-02-25T16:36:22.250 INFO:tasks.workunit.client.0.trial129.stderr:++ '[' Linux = FreeBSD ']' 2026-02-25T16:36:22.250 INFO:tasks.workunit.client.0.trial129.stderr:++ SED=sed 2026-02-25T16:36:22.250 INFO:tasks.workunit.client.0.trial129.stderr:++ AWK=awk 2026-02-25T16:36:22.251 INFO:tasks.workunit.client.0.trial129.stderr:+++ stty -a 2026-02-25T16:36:22.251 INFO:tasks.workunit.client.0.trial129.stderr:+++ head -1 2026-02-25T16:36:22.251 INFO:tasks.workunit.client.0.trial129.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-25T16:36:22.251 INFO:tasks.workunit.client.0.trial129.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T16:36:22.252 INFO:tasks.workunit.client.0.trial129.stderr:++ termwidth= 2026-02-25T16:36:22.252 INFO:tasks.workunit.client.0.trial129.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-25T16:36:22.252 INFO:tasks.workunit.client.0.trial129.stderr:++ DIFFCOLOPTS='-y ' 2026-02-25T16:36:22.252 INFO:tasks.workunit.client.0.trial129.stderr:++ KERNCORE=kernel.core_pattern 2026-02-25T16:36:22.252 INFO:tasks.workunit.client.0.trial129.stderr:++ EXTRA_OPTS= 2026-02-25T16:36:22.252 INFO:tasks.workunit.client.0.trial129.stderr:++ test '' = TESTS 2026-02-25T16:36:22.253 INFO:tasks.workunit.client.0.trial129.stderr:++ uname -m 2026-02-25T16:36:22.255 INFO:tasks.workunit.client.0.trial129.stderr:+ arch=x86_64 2026-02-25T16:36:22.255 INFO:tasks.workunit.client.0.trial129.stderr:+ case $arch in 2026-02-25T16:36:22.255 INFO:tasks.workunit.client.0.trial129.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2026-02-25T16:36:22.255 INFO:tasks.workunit.client.0.trial129.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:+ plugins=(jerasure shec lrc isa) 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:+ main test-erasure-code-plugins 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:+ local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:+ shift 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:+ shopt -s -o xtrace 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T16:36:22.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/test-erasure-code-plugins 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none ' 2026-02-25T16:36:22.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2026-02-25T16:36:22.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_blaum_roth_warning 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:TEST_ec_profile_warning 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:TEST_preload_no_warning 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:TEST_preload_no_warning_default 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:TEST_preload_warning' 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_blaum_roth_warning td/test-erasure-code-plugins 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:119: TEST_ec_profile_blaum_roth_warning: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:121: TEST_ec_profile_blaum_roth_warning: setup td/test-erasure-code-plugins 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:36:22.261 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:36:22.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:36:22.262 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:36:22.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:36:22.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:36:22.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:36:22.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:22.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:36:22.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:36:22.264 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:22.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:36:22.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:36:22.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:36:22.265 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:36:22.266 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:22.266 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:22.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:36:22.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:36:22.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:36:22.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:36:22.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:36:22.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:22.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:22.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:36:22.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:36:22.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T16:36:22.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T16:36:22.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:36:22.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:36:22.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:122: TEST_ec_profile_blaum_roth_warning: run_mon td/test-erasure-code-plugins a 2026-02-25T16:36:22.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:36:22.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:36:22.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:36:22.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:36:22.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:36:22.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:22.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:22.288 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:22.288 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:22.288 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:22.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:22.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:36:22.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:36:22.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:36:22.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:36:22.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:36:22.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:36:22.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:36:22.312 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:36:22.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:22.356 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:22.357 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:22.357 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:22.357 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:22.357 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:36:22.357 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:36:22.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:123: TEST_ec_profile_blaum_roth_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:36:22.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:36:22.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:36:22.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:36:22.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:36:22.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:36:22.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:36:22.495 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:22.495 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:22.495 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:22.495 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:22.495 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:22.495 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:22.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:36:22.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:36:22.509 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: seq 0 2 2026-02-25T16:36:22.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: for id in $(seq 0 2) 2026-02-25T16:36:22.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:125: TEST_ec_profile_blaum_roth_warning: run_osd td/test-erasure-code-plugins 0 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:36:22.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:22.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:36:22.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eb5de332-e58f-4030-9466-51a175b435d1 2026-02-25T16:36:22.517 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 eb5de332-e58f-4030-9466-51a175b435d1' 2026-02-25T16:36:22.517 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 eb5de332-e58f-4030-9466-51a175b435d1 2026-02-25T16:36:22.517 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:36:22.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAGJZ9pRVsOHxAAIz81QhyCTilqg59oMoMZug== 2026-02-25T16:36:22.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAGJZ9pRVsOHxAAIz81QhyCTilqg59oMoMZug=="}' 2026-02-25T16:36:22.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eb5de332-e58f-4030-9466-51a175b435d1 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:36:22.622 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:22.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:36:22.628 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGJZ9pRVsOHxAAIz81QhyCTilqg59oMoMZug== --osd-uuid eb5de332-e58f-4030-9466-51a175b435d1 2026-02-25T16:36:22.643 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:22.641+0000 7f158ca0a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:22.645 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:22.643+0000 7f158ca0a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:22.646 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:22.644+0000 7f158ca0a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:22.646 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:22.644+0000 7f158ca0a780 -1 bdev(0x4174c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:36:22.646 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:22.644+0000 7f158ca0a780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:36:23.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:36:23.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:36:23.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:36:23.039 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:36:23.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:36:23.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:36:23.139 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:36:23.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:36:23.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:36:23.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:36:23.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:36:23.152 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:23.151+0000 7fc22642d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:23.154 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:23.152+0000 7fc22642d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:23.155 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:23.153+0000 7fc22642d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:23.235 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:36:23.235 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:36:23.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:36:23.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:36:23.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:36:23.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:23.236 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:23.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:36:23.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:23.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:36:23.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:36:23.353 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:23.351+0000 7fc22642d780 -1 Falling back to public interface 2026-02-25T16:36:23.722 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:23.720+0000 7fc22642d780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:36:24.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:36:24.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:24.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:36:24.332 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:24.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:24.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:36:24.426 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3627202946,v1:127.0.0.1:6801/3627202946] [v2:127.0.0.1:6802/3627202946,v1:127.0.0.1:6803/3627202946] exists,up eb5de332-e58f-4030-9466-51a175b435d1 2026-02-25T16:36:24.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:36:24.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:36:24.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:36:24.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: for id in $(seq 0 2) 2026-02-25T16:36:24.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:125: TEST_ec_profile_blaum_roth_warning: run_osd td/test-erasure-code-plugins 1 2026-02-25T16:36:24.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:24.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/1 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:36:24.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:36:24.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/1 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5058eb43-4925-4934-a7e5-daaa8606024d 2026-02-25T16:36:24.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5058eb43-4925-4934-a7e5-daaa8606024d' 2026-02-25T16:36:24.434 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 5058eb43-4925-4934-a7e5-daaa8606024d 2026-02-25T16:36:24.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:36:24.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIJZ9pPqUCGhAAeIybgPLqmlR2LrjYlhXewg== 2026-02-25T16:36:24.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIJZ9pPqUCGhAAeIybgPLqmlR2LrjYlhXewg=="}' 2026-02-25T16:36:24.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5058eb43-4925-4934-a7e5-daaa8606024d -i td/test-erasure-code-plugins/1/new.json 2026-02-25T16:36:24.538 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:24.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/1/new.json 2026-02-25T16:36:24.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIJZ9pPqUCGhAAeIybgPLqmlR2LrjYlhXewg== --osd-uuid 5058eb43-4925-4934-a7e5-daaa8606024d 2026-02-25T16:36:24.559 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:24.557+0000 7f03e7649780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:24.560 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:24.558+0000 7f03e7649780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:24.561 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:24.559+0000 7f03e7649780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:24.562 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:24.560+0000 7f03e7649780 -1 bdev(0x4008c00 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:36:24.562 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:24.560+0000 7f03e7649780 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2026-02-25T16:36:24.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2026-02-25T16:36:24.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:36:24.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:36:24.933 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:36:24.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:36:25.244 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:36:25.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:36:25.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:36:25.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:36:25.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:36:25.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:36:25.256 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:25.255+0000 7fd5be047780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:25.258 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:25.256+0000 7fd5be047780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:25.259 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:25.257+0000 7fd5be047780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:25.400 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:25.397+0000 7fd5be047780 -1 Falling back to public interface 2026-02-25T16:36:25.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:36:25.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:25.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:36:25.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:36:25.778 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:25.775+0000 7fd5be047780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:36:26.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:36:26.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:26.690 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:26.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:36:26.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:26.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:36:26.914 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1652864318,v1:127.0.0.1:6811/1652864318] [v2:127.0.0.1:6812/1652864318,v1:127.0.0.1:6813/1652864318] exists,up 5058eb43-4925-4934-a7e5-daaa8606024d 2026-02-25T16:36:26.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:36:26.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:36:26.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:36:26.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:124: TEST_ec_profile_blaum_roth_warning: for id in $(seq 0 2) 2026-02-25T16:36:26.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:125: TEST_ec_profile_blaum_roth_warning: run_osd td/test-erasure-code-plugins 2 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/2 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:26.916 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/2 2026-02-25T16:36:26.917 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:36:26.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a2b0a85-3d7f-44f7-83a6-a51344d4cd58 2026-02-25T16:36:26.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4a2b0a85-3d7f-44f7-83a6-a51344d4cd58' 2026-02-25T16:36:26.918 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 4a2b0a85-3d7f-44f7-83a6-a51344d4cd58 2026-02-25T16:36:26.918 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:36:26.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAKJZ9pxpkzNxAAnDHifFqZlYxpVQj52hZdBA== 2026-02-25T16:36:26.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAKJZ9pxpkzNxAAnDHifFqZlYxpVQj52hZdBA=="}' 2026-02-25T16:36:26.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a2b0a85-3d7f-44f7-83a6-a51344d4cd58 -i td/test-erasure-code-plugins/2/new.json 2026-02-25T16:36:27.151 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:36:27.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/2/new.json 2026-02-25T16:36:27.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKJZ9pxpkzNxAAnDHifFqZlYxpVQj52hZdBA== --osd-uuid 4a2b0a85-3d7f-44f7-83a6-a51344d4cd58 2026-02-25T16:36:27.172 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.170+0000 7fae021d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:27.174 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.172+0000 7fae021d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:27.174 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.173+0000 7fae021d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:27.175 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.173+0000 7fae021d8780 -1 bdev(0x432ac00 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:36:27.175 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.173+0000 7fae021d8780 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2026-02-25T16:36:27.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2026-02-25T16:36:27.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:36:27.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:36:27.535 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:36:27.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:36:27.845 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:36:27.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:36:27.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:36:27.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:36:27.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:36:27.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:36:27.857 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.856+0000 7f4d1a5d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:27.859 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.857+0000 7f4d1a5d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:27.859 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:27.858+0000 7f4d1a5d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:28.050 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:28.048+0000 7f4d1a5d8780 -1 Falling back to public interface 2026-02-25T16:36:28.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:36:28.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:36:28.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:36:28.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:36:28.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:36:28.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:28.071 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:28.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:36:28.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:28.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:36:28.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:36:28.402 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:28.399+0000 7f4d1a5d8780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:36:29.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:36:29.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:29.296 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:29.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:36:29.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:29.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:36:29.523 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/663178508,v1:127.0.0.1:6819/663178508] [v2:127.0.0.1:6820/663178508,v1:127.0.0.1:6821/663178508] exists,up 4a2b0a85-3d7f-44f7-83a6-a51344d4cd58 2026-02-25T16:36:29.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:36:29.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:36:29.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:36:29.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:127: TEST_ec_profile_blaum_roth_warning: create_rbd_pool 2026-02-25T16:36:29.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T16:36:29.738 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' does not exist 2026-02-25T16:36:29.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T16:36:29.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:36:29.969 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:36:29.976 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:36:30.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T16:36:31.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:128: TEST_ec_profile_blaum_roth_warning: wait_for_clean 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:36:31.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:36:31.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:36:31.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:36:31.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:36:31.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:36:31.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:36:31.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:36:31.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:36:31.562 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:36:31.563 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:36:31.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:36:31.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:36:31.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:36:31.628 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-25T16:36:31.628 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-25T16:36:31.628 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484' 2026-02-25T16:36:31.628 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:36:31.628 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:36:31.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-25T16:36:31.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-25T16:36:31.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484 1-38654705668' 2026-02-25T16:36:31.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:36:31.690 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:36:31.751 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-25T16:36:31.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-25T16:36:31.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484 1-38654705668 2-55834574852' 2026-02-25T16:36:31.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:36:31.752 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836484 2026-02-25T16:36:31.752 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:36:31.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:36:31.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-25T16:36:31.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:36:31.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836484 2026-02-25T16:36:31.754 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 21474836484 2026-02-25T16:36:31.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-25T16:36:31.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:36:31.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-25T16:36:31.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:36:32.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:36:32.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:36:33.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-25T16:36:33.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:36:33.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705668 2026-02-25T16:36:33.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:36:33.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:36:33.207 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-25T16:36:33.208 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:36:33.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705668 2026-02-25T16:36:33.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-25T16:36:33.209 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 38654705668 2026-02-25T16:36:33.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:36:33.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705669 -lt 38654705668 2026-02-25T16:36:33.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:36:33.431 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574852 2026-02-25T16:36:33.431 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:36:33.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:36:33.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-25T16:36:33.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:36:33.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574852 2026-02-25T16:36:33.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-25T16:36:33.435 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 55834574852 2026-02-25T16:36:33.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:36:33.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-25T16:36:33.660 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:36:33.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:36:33.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:36:33.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T16:36:33.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:36:33.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:36:33.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:36:33.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:36:33.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:36:33.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:36:33.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:36:34.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T16:36:34.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:36:34.195 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:36:34.195 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:36:34.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T16:36:34.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:36:34.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:36:34.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:130: TEST_ec_profile_blaum_roth_warning: echo 'Starting test for blaum-roth profile health warning' 2026-02-25T16:36:34.512 INFO:tasks.workunit.client.0.trial129.stdout:Starting test for blaum-roth profile health warning 2026-02-25T16:36:34.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:133: TEST_ec_profile_blaum_roth_warning: ceph osd erasure-code-profile set prof- plugin=jerasure k=3 m=1 technique=blaum_roth w=7 --yes-i-really-mean-it --force 2026-02-25T16:36:34.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:135: TEST_ec_profile_blaum_roth_warning: ceph osd erasure-code-profile set test crush-device-class= crush-failure-domain=osd crush-num-failure-domains=0 crush-osds-per-failure-domain=0 crush-root=default k=4 l=3 m=2 name=lrcprofile plugin=lrc 2026-02-25T16:36:35.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:136: TEST_ec_profile_blaum_roth_warning: get_asok_path mon.a 2026-02-25T16:36:35.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:35.107 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:35.107 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:35.107 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:35.107 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:35.107 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:35.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:136: TEST_ec_profile_blaum_roth_warning: CEPH_ARGS= 2026-02-25T16:36:35.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:136: TEST_ec_profile_blaum_roth_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:36:35.145 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:36:35.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:137: TEST_ec_profile_blaum_roth_warning: sleep 10 2026-02-25T16:36:45.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:138: TEST_ec_profile_blaum_roth_warning: grep -F '1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:45.153 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:37.309+0000 7f3d32df6640 0 log_channel(cluster) log [WRN] : Health check failed: 1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption (BLAUM_ROTH_W_IS_NOT_PRIME) 2026-02-25T16:36:45.153 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:37.309+0000 7f3d305f1640 10 mon.a@0(leader).log v77 logging 2026-02-25T16:36:37.310047+0000 mon.a (mon.0) 141 : cluster [WRN] Health check failed: 1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption (BLAUM_ROTH_W_IS_NOT_PRIME) 2026-02-25T16:36:45.153 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.153 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:37.359+0000 7f3d2ddec640 7 mon.a@0(leader).log v78 update_from_paxos applying incremental log 78 2026-02-25T16:36:37.310047+0000 mon.a (mon.0) 141 : cluster [WRN] Health check failed: 1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption (BLAUM_ROTH_W_IS_NOT_PRIME) 2026-02-25T16:36:45.153 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.153 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stdout: "message": "1 or more EC profiles have a w value such that w+1 is not prime. This can result in data corruption", 2026-02-25T16:36:45.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:139: TEST_ec_profile_blaum_roth_warning: grep -F 'w+1=8 for the EC profile prof- is not prime and could lead to data corruption' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.156 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.157 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.157 INFO:tasks.workunit.client.0.trial129.stdout: "message": "w+1=8 for the EC profile prof- is not prime and could lead to data corruption" 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:141: TEST_ec_profile_blaum_roth_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:36:45.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:36:45.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:36:45.264 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:36:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:36:45.265 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:36:45.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:36:45.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:36:45.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:36:45.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:45.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:36:45.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:36:45.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:45.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:36:45.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:36:45.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:36:45.276 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:36:45.277 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.277 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:142: TEST_ec_profile_blaum_roth_warning: return 0 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:36:45.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:36:45.281 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:36:45.281 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:36:45.281 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:36:45.281 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:36:45.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:36:45.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:36:45.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:36:45.283 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:45.283 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:36:45.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:36:45.284 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:45.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:36:45.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:36:45.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:36:45.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:36:45.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:36:45.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:36:45.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:36:45.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:36:45.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:36:45.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:36:45.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:36:45.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:36:45.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.308 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:36:45.308 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:45.309 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:45.309 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:45.309 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.309 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.309 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:45.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:36:45.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:36:45.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:36:45.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:36:45.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:36:45.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:36:45.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:36:45.333 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:36:45.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:36:45.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:36:45.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:36:45.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:36:45.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:36:45.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:36:45.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:36:45.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:36:45.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:36:45.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:36:45.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:36:45.517 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:36:45.517 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:45.517 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:45.517 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:45.517 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.517 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.517 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:45.518 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:36:45.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:36:45.532 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2026-02-25T16:36:45.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:36:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:45.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:36:45.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:36:45.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=18562764-48cf-414d-9a66-4a56091862d2 2026-02-25T16:36:45.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 18562764-48cf-414d-9a66-4a56091862d2' 2026-02-25T16:36:45.540 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 18562764-48cf-414d-9a66-4a56091862d2 2026-02-25T16:36:45.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:36:45.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdJZ9pzCd0IBAAFAfEeCAC+qPu3tGTPugvnQ== 2026-02-25T16:36:45.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdJZ9pzCd0IBAAFAfEeCAC+qPu3tGTPugvnQ=="}' 2026-02-25T16:36:45.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 18562764-48cf-414d-9a66-4a56091862d2 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:36:45.646 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:45.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:36:45.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdJZ9pzCd0IBAAFAfEeCAC+qPu3tGTPugvnQ== --osd-uuid 18562764-48cf-414d-9a66-4a56091862d2 2026-02-25T16:36:45.667 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:45.665+0000 7fab5a46d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:45.669 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:45.667+0000 7fab5a46d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:45.670 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:45.668+0000 7fab5a46d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:45.670 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:45.668+0000 7fab5a46d780 -1 bdev(0x31fcc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:36:45.670 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:45.668+0000 7fab5a46d780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:36:46.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:36:46.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:36:46.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:36:46.096 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:36:46.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:36:46.198 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:36:46.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:36:46.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:36:46.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:36:46.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:36:46.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:36:46.211 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:46.209+0000 7fddecc31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:46.212 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:46.210+0000 7fddecc31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:46.213 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:46.211+0000 7fddecc31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:46.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:36:46.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:36:46.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:36:46.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:36:46.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:36:46.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:46.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:36:46.297 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:46.298 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:46.298 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:36:46.388 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:46.385+0000 7fddecc31780 -1 Falling back to public interface 2026-02-25T16:36:46.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:36:46.754 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:46.752+0000 7fddecc31780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:36:47.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:36:47.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:47.392 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:47.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:36:47.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:47.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:36:47.485 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4186156523,v1:127.0.0.1:6801/4186156523] [v2:127.0.0.1:6802/4186156523,v1:127.0.0.1:6803/4186156523] exists,up 18562764-48cf-414d-9a66-4a56091862d2 2026-02-25T16:36:47.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:36:47.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:36:47.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/1 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:36:47.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:47.491 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:36:47.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/1 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:36:47.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7ba197bc-47d4-43cd-82d6-ce02fbaa478b 2026-02-25T16:36:47.494 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 7ba197bc-47d4-43cd-82d6-ce02fbaa478b 2026-02-25T16:36:47.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7ba197bc-47d4-43cd-82d6-ce02fbaa478b' 2026-02-25T16:36:47.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:36:47.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAfJZ9p/wuQHRAAwP077iXTAi6DMZpoiZ+dyQ== 2026-02-25T16:36:47.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAfJZ9p/wuQHRAAwP077iXTAi6DMZpoiZ+dyQ=="}' 2026-02-25T16:36:47.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7ba197bc-47d4-43cd-82d6-ce02fbaa478b -i td/test-erasure-code-plugins/1/new.json 2026-02-25T16:36:47.599 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:47.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/1/new.json 2026-02-25T16:36:47.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfJZ9p/wuQHRAAwP077iXTAi6DMZpoiZ+dyQ== --osd-uuid 7ba197bc-47d4-43cd-82d6-ce02fbaa478b 2026-02-25T16:36:47.619 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:47.617+0000 7efec1245780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:47.620 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:47.618+0000 7efec1245780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:47.621 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:47.619+0000 7efec1245780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:47.622 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:47.619+0000 7efec1245780 -1 bdev(0x40b8c00 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:36:47.622 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:47.619+0000 7efec1245780 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2026-02-25T16:36:47.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2026-02-25T16:36:47.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:36:47.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:36:47.988 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:36:47.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:36:48.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:36:48.299 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:36:48.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:36:48.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:36:48.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:36:48.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:36:48.312 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:48.310+0000 7fe8a33d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:48.313 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:48.311+0000 7fe8a33d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:48.314 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:48.312+0000 7fe8a33d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:48.483 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:48.481+0000 7fe8a33d8780 -1 Falling back to public interface 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:36:48.524 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:48.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:48.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:36:48.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:36:48.876 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:48.873+0000 7fe8a33d8780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:36:49.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:36:49.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:49.745 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:49.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:36:49.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:49.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:36:49.973 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2142643768,v1:127.0.0.1:6811/2142643768] [v2:127.0.0.1:6812/2142643768,v1:127.0.0.1:6813/2142643768] exists,up 7ba197bc-47d4-43cd-82d6-ce02fbaa478b 2026-02-25T16:36:49.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:36:49.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:36:49.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/2 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:36:49.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:49.979 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:36:49.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/2 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ddea6fcd-3a2e-424f-8d27-cefc3e5d3930 2026-02-25T16:36:49.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ddea6fcd-3a2e-424f-8d27-cefc3e5d3930' 2026-02-25T16:36:49.982 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 ddea6fcd-3a2e-424f-8d27-cefc3e5d3930 2026-02-25T16:36:49.982 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:36:49.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhJZ9pvTGvOhAAJeBA30FxIZRPdyIp9/b96Q== 2026-02-25T16:36:49.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhJZ9pvTGvOhAAJeBA30FxIZRPdyIp9/b96Q=="}' 2026-02-25T16:36:49.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ddea6fcd-3a2e-424f-8d27-cefc3e5d3930 -i td/test-erasure-code-plugins/2/new.json 2026-02-25T16:36:50.210 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:36:50.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/2/new.json 2026-02-25T16:36:50.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhJZ9pvTGvOhAAJeBA30FxIZRPdyIp9/b96Q== --osd-uuid ddea6fcd-3a2e-424f-8d27-cefc3e5d3930 2026-02-25T16:36:50.232 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.230+0000 7f2cc5e12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:50.233 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.231+0000 7f2cc5e12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:50.234 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.232+0000 7f2cc5e12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:50.235 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.232+0000 7f2cc5e12780 -1 bdev(0x24b6c00 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:36:50.235 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.232+0000 7f2cc5e12780 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2026-02-25T16:36:50.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2026-02-25T16:36:50.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:36:50.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:36:50.599 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:36:50.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:36:50.915 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:36:50.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:36:50.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:36:50.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:36:50.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:36:50.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:36:50.929 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.926+0000 7f0ccf054780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:50.930 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.928+0000 7f0ccf054780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:50.931 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:50.929+0000 7f0ccf054780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:36:51.074 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:51.071+0000 7f0ccf054780 -1 Falling back to public interface 2026-02-25T16:36:51.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:36:51.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:36:51.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:36:51.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:36:51.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:36:51.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:51.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:36:51.139 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:36:51.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:51.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:36:51.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:36:51.397 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:36:51.395+0000 7f0ccf054780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:36:52.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:36:52.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:36:52.366 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:36:52.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:36:52.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:36:52.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:36:52.590 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3918359824,v1:127.0.0.1:6819/3918359824] [v2:127.0.0.1:6820/3918359824,v1:127.0.0.1:6821/3918359824] exists,up ddea6fcd-3a2e-424f-8d27-cefc3e5d3930 2026-02-25T16:36:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:36:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:36:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:36:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2026-02-25T16:36:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T16:36:52.805 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' does not exist 2026-02-25T16:36:52.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T16:36:52.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:36:53.059 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:36:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:36:54.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:36:54.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:36:54.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:36:54.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:36:54.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:36:54.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:36:54.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:36:54.430 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:36:54.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:36:54.659 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:36:54.659 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:36:54.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:36:54.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:36:54.659 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:36:54.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-25T16:36:54.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-25T16:36:54.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484' 2026-02-25T16:36:54.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:36:54.722 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:36:54.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-25T16:36:54.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-25T16:36:54.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-25T16:36:54.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:36:54.785 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:36:54.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-25T16:36:54.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-25T16:36:54.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-60129542147' 2026-02-25T16:36:54.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:36:54.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836484 2026-02-25T16:36:54.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:36:54.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:36:54.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-25T16:36:54.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:36:54.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836484 2026-02-25T16:36:54.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-25T16:36:54.850 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 21474836484 2026-02-25T16:36:54.850 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:36:55.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-25T16:36:55.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:36:56.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:36:56.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:36:56.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-25T16:36:56.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:36:56.303 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672963 2026-02-25T16:36:56.303 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:36:56.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:36:56.304 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-25T16:36:56.304 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:36:56.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672963 2026-02-25T16:36:56.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-25T16:36:56.305 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 42949672963 2026-02-25T16:36:56.306 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:36:56.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672964 -lt 42949672963 2026-02-25T16:36:56.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:36:56.528 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542147 2026-02-25T16:36:56.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:36:56.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:36:56.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-25T16:36:56.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:36:56.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542147 2026-02-25T16:36:56.531 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 60129542147 2026-02-25T16:36:56.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-25T16:36:56.532 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:36:56.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-25T16:36:56.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:36:56.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:36:56.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:36:57.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T16:36:57.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:36:57.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:36:57.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:36:57.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:36:57.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:36:57.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:36:57.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:36:57.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T16:36:57.294 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:36:57.294 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:36:57.294 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:36:57.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T16:36:57.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:36:57.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:36:57.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-25T16:36:57.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2026-02-25T16:36:57.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T16:36:57.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:36:57.935 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:36:57.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:57.941 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:57.768+0000 7f1365df1640 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:57.941 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:57.885+0000 7f1365df1640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:57.941 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:57.885+0000 7f1365df1640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:57.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-25T16:36:57.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2026-02-25T16:36:58.225 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T16:36:58.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:36:58.264 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:36:58.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:58.271 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.098+0000 7f1365df1640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:58.271 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.214+0000 7f1365df1640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:58.271 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.214+0000 7f1365df1640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:58.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-25T16:36:58.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T16:36:58.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:36:58.601 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:36:58.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:58.607 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.432+0000 7f1365df1640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:58.607 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.551+0000 7f1365df1640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:58.607 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.551+0000 7f1365df1640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-25T16:36:58.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-25T16:36:58.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:58.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T16:36:58.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:36:58.933 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:36:58.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:58.941 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.765+0000 7f1365df1640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-25T16:36:58.941 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.884+0000 7f1365df1640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-25T16:36:58.941 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:58.884+0000 7f1365df1640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-25T16:36:58.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-25T16:36:58.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2026-02-25T16:36:59.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T16:36:59.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:36:59.265 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:36:59.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:59.270 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:59.099+0000 7f1365df1640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-25T16:36:59.270 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:59.216+0000 7f1365df1640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-25T16:36:59.271 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:59.216+0000 7f1365df1640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-25T16:36:59.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-25T16:36:59.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-25T16:36:59.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:36:59.597 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:36:59.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:36:59.603 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:59.430+0000 7f1365df1640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-25T16:36:59.603 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:59.548+0000 7f1365df1640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-25T16:36:59.603 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:36:59.548+0000 7f1365df1640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-25T16:36:59.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:36:59.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:59.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:36:59.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:36:59.604 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:36:59.604 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:36:59.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:36:59.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:36:59.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:36:59.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:36:59.711 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:36:59.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:36:59.712 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:36:59.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:36:59.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:36:59.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:36:59.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:59.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:36:59.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:36:59.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:59.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:36:59.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:36:59.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:36:59.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:36:59.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:36:59.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:36:59.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:36:59.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T16:36:59.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2026-02-25T16:36:59.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:36:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:36:59.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:36:59.727 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:36:59.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:36:59.727 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:36:59.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:36:59.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:36:59.728 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:36:59.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:59.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:36:59.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:36:59.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:36:59.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:36:59.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:36:59.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:36:59.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:36:59.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:36:59.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:36:59.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:36:59.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:36:59.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:36:59.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:36:59.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:36:59.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:36:59.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2026-02-25T16:36:59.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:36:59.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:59.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:59.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:59.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:59.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2026-02-25T16:36:59.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:36:59.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:36:59.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:36:59.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:36:59.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:36:59.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:59.822 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:59.823 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:59.823 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.823 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.823 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:59.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:36:59.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:36:59.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:36:59.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:36:59.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:36:59.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:36:59.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:36:59.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:36:59.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:36:59.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:36:59.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:36:59.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:36:59.961 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:36:59.961 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.961 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.961 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:36:59.961 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:36:59.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:36:59.977 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T16:36:59.977 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:36:59.977 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:36:59.977 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:36:59.977 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:36:59.977 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:36:59.977 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:36:59.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:36:59.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:00.014 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:00.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2026-02-25T16:37:00.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:00.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:00.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:00.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f6f693f1-dc62-488f-a62c-20a866fddc6c 2026-02-25T16:37:00.024 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 f6f693f1-dc62-488f-a62c-20a866fddc6c 2026-02-25T16:37:00.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f6f693f1-dc62-488f-a62c-20a866fddc6c' 2026-02-25T16:37:00.024 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:00.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAsJZ9pheOkARAAu5yNuk+BmgnKZO3i3FkLqg== 2026-02-25T16:37:00.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAsJZ9pheOkARAAu5yNuk+BmgnKZO3i3FkLqg=="}' 2026-02-25T16:37:00.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f6f693f1-dc62-488f-a62c-20a866fddc6c -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:00.129 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:00.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:00.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQAsJZ9pheOkARAAu5yNuk+BmgnKZO3i3FkLqg== --osd-uuid f6f693f1-dc62-488f-a62c-20a866fddc6c 2026-02-25T16:37:00.151 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.149+0000 7fd467653780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:00.152 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.150+0000 7fd467653780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:00.153 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.151+0000 7fd467653780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:00.154 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.151+0000 7fd467653780 -1 bdev(0x2b50c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:00.154 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.151+0000 7fd467653780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:00.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:00.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:00.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:00.560 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:00.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:00.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:00.662 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:00.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2026-02-25T16:37:00.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:00.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:00.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:00.676 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.673+0000 7f804c045780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:00.677 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.675+0000 7f804c045780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:00.678 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.676+0000 7f804c045780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:00.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:00.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:00.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:00.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:00.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:00.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:00.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:00.758 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:00.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:00.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:00.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:00.882 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:00.880+0000 7f804c045780 -1 Falling back to public interface 2026-02-25T16:37:01.291 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:01.288+0000 7f804c045780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:01.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:01.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:01.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:01.851 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:01.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:01.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:01.946 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1110140340,v1:127.0.0.1:6801/1110140340] [v2:127.0.0.1:6802/1110140340,v1:127.0.0.1:6803/1110140340] exists,up f6f693f1-dc62-488f-a62c-20a866fddc6c 2026-02-25T16:37:01.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:01.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:01.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:01.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:37:01.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:01.991 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:01.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:01.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:01.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:01.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:01.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:01.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:01.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:01.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:01.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:01.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:01.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:02.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:02.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:02.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:02.107 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:02.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:02.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:02.108 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:02.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:02.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:02.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:02.110 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:02.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:02.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:02.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:02.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:02.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:02.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:02.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:02.118 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:02.118 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:02.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:02.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:02.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:02.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:02.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:02.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:02.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:02.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:02.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:02.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:02.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:02.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:02.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:02.122 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:02.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:02.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:02.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:02.124 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:02.125 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.125 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:02.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:02.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:02.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:02.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:02.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:02.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:02.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:02.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:02.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:02.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2026-02-25T16:37:02.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:02.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:02.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:02.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:02.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:02.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2026-02-25T16:37:02.146 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:02.146 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:02.146 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:02.146 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:02.146 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.147 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:02.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2026-02-25T16:37:02.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:02.171 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.172 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.172 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:02.172 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:02.172 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:02.215 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:02.216 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:02.216 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.216 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.216 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:02.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:02.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:02.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:02.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:02.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:02.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:02.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:02.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:02.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:02.351 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:02.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:02.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:02.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:02.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:02.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:02.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:37:02.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:02.405 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:02.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:02.415 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:02.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ec03875a-95c2-45c6-9cdb-a7f79afcbd7f 2026-02-25T16:37:02.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ec03875a-95c2-45c6-9cdb-a7f79afcbd7f' 2026-02-25T16:37:02.416 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 ec03875a-95c2-45c6-9cdb-a7f79afcbd7f 2026-02-25T16:37:02.416 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:02.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAuJZ9purESGRAALmVww7n2bvasHsORDHja0A== 2026-02-25T16:37:02.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAuJZ9purESGRAALmVww7n2bvasHsORDHja0A=="}' 2026-02-25T16:37:02.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ec03875a-95c2-45c6-9cdb-a7f79afcbd7f -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:02.523 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:02.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:02.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQAuJZ9purESGRAALmVww7n2bvasHsORDHja0A== --osd-uuid ec03875a-95c2-45c6-9cdb-a7f79afcbd7f 2026-02-25T16:37:02.544 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:02.543+0000 7fad3ee42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:02.546 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:02.544+0000 7fad3ee42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:02.547 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:02.545+0000 7fad3ee42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:02.547 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:02.545+0000 7fad3ee42780 -1 bdev(0x3af8c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:02.547 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:02.545+0000 7fad3ee42780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:02.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:02.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:02.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:02.927 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:02.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:03.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:03.027 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:03.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2026-02-25T16:37:03.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:03.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:03.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:03.040 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:03.038+0000 7f94fec5e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:03.041 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:03.039+0000 7f94fec5e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:03.042 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:03.040+0000 7f94fec5e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:03.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:03.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:03.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:03.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:03.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:03.122 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:03.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:03.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:03.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:03.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:03.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:03.237 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:03.235+0000 7f94fec5e780 -1 Falling back to public interface 2026-02-25T16:37:03.626 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:03.624+0000 7f94fec5e780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:04.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:04.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:04.216 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:04.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:04.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:04.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:04.311 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/62495654,v1:127.0.0.1:6801/62495654] [v2:127.0.0.1:6802/62495654,v1:127.0.0.1:6803/62495654] exists,up ec03875a-95c2-45c6-9cdb-a7f79afcbd7f 2026-02-25T16:37:04.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:04.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:04.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:04.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T16:37:04.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:04.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:04.313 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:04.314 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.314 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.314 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:04.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:37:04.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:04.358 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:04.363 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:04.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:04.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:04.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:04.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:04.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:04.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:04.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:04.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:04.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:04.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:04.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:04.474 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:04.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:04.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:04.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:04.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:04.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:04.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:04.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:04.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:04.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:04.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:04.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:04.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:04.483 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:04.483 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T16:37:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:04.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:04.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:04.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:04.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:04.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2026-02-25T16:37:04.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:04.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:04.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:04.515 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:04.515 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.515 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.516 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:04.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2026-02-25T16:37:04.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:04.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:04.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:04.541 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:04.541 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:04.541 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:04.541 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:04.541 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:04.541 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:04.542 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:04.542 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.542 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.542 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:04.542 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:04.542 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:04.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:04.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:04.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:04.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:04.584 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:04.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:04.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:04.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:04.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:04.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:04.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:04.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:04.721 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:04.721 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:04.721 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:04.721 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:04.721 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.722 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.722 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:04.722 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:04.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:04.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T16:37:04.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:04.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:04.738 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:04.739 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.739 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.739 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:04.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:37:04.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:04.776 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:04.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2026-02-25T16:37:04.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:04.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:04.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:04.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:04.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:04.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:04.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:04.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:04.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:04.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:04.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:04.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:04.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:04.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:04.790 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:04.790 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:04.790 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:04.791 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:04.791 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:04.791 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:04.791 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:04.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:04.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:04.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:04.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:04.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:04.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:04.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:04.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:04.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:04.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:04.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:04.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2026-02-25T16:37:04.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:04.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:04.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2c0e9582-e57c-45f5-abda-66e1680deb2f 2026-02-25T16:37:04.794 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 2c0e9582-e57c-45f5-abda-66e1680deb2f 2026-02-25T16:37:04.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2c0e9582-e57c-45f5-abda-66e1680deb2f' 2026-02-25T16:37:04.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:04.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAwJZ9pg8w2LxAAUrhebwIOT2vhUF/iYZwPaw== 2026-02-25T16:37:04.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAwJZ9pg8w2LxAAUrhebwIOT2vhUF/iYZwPaw=="}' 2026-02-25T16:37:04.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2c0e9582-e57c-45f5-abda-66e1680deb2f -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:04.892 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:04.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:04.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQAwJZ9pg8w2LxAAUrhebwIOT2vhUF/iYZwPaw== --osd-uuid 2c0e9582-e57c-45f5-abda-66e1680deb2f 2026-02-25T16:37:04.915 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:04.913+0000 7f5effe66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:04.916 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:04.914+0000 7f5effe66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:04.916 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:04.915+0000 7f5effe66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:04.917 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:04.915+0000 7f5effe66780 -1 bdev(0x27b8c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:04.917 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:04.915+0000 7f5effe66780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:05.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:05.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:05.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:05.295 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:05.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:05.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:05.393 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:05.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2026-02-25T16:37:05.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:05.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:05.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:05.407 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:05.405+0000 7f405e20c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:05.408 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:05.407+0000 7f405e20c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:05.409 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:05.407+0000 7f405e20c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:05.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:05.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:05.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:05.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:05.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:05.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:05.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:05.494 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:05.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:05.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:05.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:05.611 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:05.610+0000 7f405e20c780 -1 Falling back to public interface 2026-02-25T16:37:06.022 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:06.020+0000 7f405e20c780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:06.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:06.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:06.589 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:06.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:06.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:06.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:06.683 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/83933215,v1:127.0.0.1:6801/83933215] [v2:127.0.0.1:6802/83933215,v1:127.0.0.1:6803/83933215] exists,up 2c0e9582-e57c-45f5-abda-66e1680deb2f 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:06.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:37:06.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:06.729 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:06.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:06.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:06.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:06.842 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:06.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:06.843 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:06.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:06.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:06.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:06.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:06.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:06.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:06.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:06.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:06.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:06.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:06.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:06.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:06.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:06.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:06.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:06.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:06.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-25T16:37:06.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:06.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:06.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:06.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:06.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:06.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:06.854 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:06.854 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:06.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:06.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:06.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:06.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:06.855 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:06.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:06.856 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:06.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:06.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:06.857 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:06.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:06.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:06.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:06.859 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:06.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:06.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:06.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:06.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:06.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:06.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:06.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:06.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:06.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:06.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:06.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:06.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:06.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:06.864 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:06.864 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:06.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:06.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:06.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2026-02-25T16:37:06.882 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:06.882 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:06.882 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:06.882 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:06.883 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:06.883 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:06.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:06.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2026-02-25T16:37:06.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:06.906 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:06.907 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:06.907 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:06.907 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:06.907 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:06.908 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:06.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:06.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:06.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:06.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:06.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:06.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:06.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:06.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:06.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:06.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:06.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:06.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:06.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:06.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:07.088 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:07.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:07.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:07.089 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:07.089 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:07.089 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:07.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:07.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:07.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:07.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:37:07.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:07.144 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:07.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:07.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:07.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:07.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:07.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2026-02-25T16:37:07.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:07.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:07.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e57a84d6-8f99-45ea-9b60-76ca2a0b8582 2026-02-25T16:37:07.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e57a84d6-8f99-45ea-9b60-76ca2a0b8582' 2026-02-25T16:37:07.154 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 e57a84d6-8f99-45ea-9b60-76ca2a0b8582 2026-02-25T16:37:07.155 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:07.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzJZ9pRDNxCRAAuFz0ULw6CiTpmIbs0DVJow== 2026-02-25T16:37:07.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzJZ9pRDNxCRAAuFz0ULw6CiTpmIbs0DVJow=="}' 2026-02-25T16:37:07.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e57a84d6-8f99-45ea-9b60-76ca2a0b8582 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:07.258 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:07.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:07.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQAzJZ9pRDNxCRAAuFz0ULw6CiTpmIbs0DVJow== --osd-uuid e57a84d6-8f99-45ea-9b60-76ca2a0b8582 2026-02-25T16:37:07.280 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.279+0000 7fb28942f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:07.282 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.280+0000 7fb28942f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:07.283 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.281+0000 7fb28942f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:07.284 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.282+0000 7fb28942f780 -1 bdev(0x2d36c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:07.284 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.282+0000 7fb28942f780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:07.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:07.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:07.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:07.675 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:07.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:07.776 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:07.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:07.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2026-02-25T16:37:07.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:07.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:07.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:07.789 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.787+0000 7f6f1bbd8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:07.790 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.789+0000 7f6f1bbd8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:07.791 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.790+0000 7f6f1bbd8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:07.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:07.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:07.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:07.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:07.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:07.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:07.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:07.872 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:07.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:07.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:07.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:07.994 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:07.993+0000 7f6f1bbd8780 -1 Falling back to public interface 2026-02-25T16:37:08.425 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:08.424+0000 7f6f1bbd8780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:08.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:08.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:08.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:08.967 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:08.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:08.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:09.060 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1904437764,v1:127.0.0.1:6801/1904437764] [v2:127.0.0.1:6802/1904437764,v1:127.0.0.1:6803/1904437764] exists,up e57a84d6-8f99-45ea-9b60-76ca2a0b8582 2026-02-25T16:37:09.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.061 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.062 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-25T16:37:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:09.106 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:09.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:09.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:09.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:09.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:09.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:09.113 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:09.113 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:09.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:09.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:09.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:09.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:09.220 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:09.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:09.222 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:09.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:09.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:09.223 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:09.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:09.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:09.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:09.225 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:09.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:09.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:09.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:09.229 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:09.230 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.230 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:09.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:09.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:09.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2026-02-25T16:37:09.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T16:37:09.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2026-02-25T16:37:09.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:09.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:09.234 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:09.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:09.234 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:09.235 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:09.235 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:09.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:09.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:09.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:09.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:09.237 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:09.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:09.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:09.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:09.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:09.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:09.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:09.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:09.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:09.241 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:09.241 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:09.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:09.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:09.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.260 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:09.261 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:09.261 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:09.261 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:09.261 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.261 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.261 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:09.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:37:09.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:09.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.287 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.288 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:09.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:09.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:09.329 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.330 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.331 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:09.331 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:09.331 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:09.374 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2026-02-25T16:37:09.374 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:09.374 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:09.375 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:09.375 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.375 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:09.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2026-02-25T16:37:09.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:09.410 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:09.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:09.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:09.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:09.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:09.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:09.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:09.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:09.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:09.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:09.514 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:09.514 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.514 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:09.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:09.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:09.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2026-02-25T16:37:09.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:09.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:09.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:09.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:09.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:09.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a7bb2d2-77b5-43cf-9a40-0b63b3f6e7f9 2026-02-25T16:37:09.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4a7bb2d2-77b5-43cf-9a40-0b63b3f6e7f9' 2026-02-25T16:37:09.537 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 4a7bb2d2-77b5-43cf-9a40-0b63b3f6e7f9 2026-02-25T16:37:09.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:09.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1JZ9pB7g8IBAAvw0XwbgUx8iEqmhQuDmmPQ== 2026-02-25T16:37:09.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1JZ9pB7g8IBAAvw0XwbgUx8iEqmhQuDmmPQ=="}' 2026-02-25T16:37:09.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a7bb2d2-77b5-43cf-9a40-0b63b3f6e7f9 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:09.644 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:09.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:09.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1JZ9pB7g8IBAAvw0XwbgUx8iEqmhQuDmmPQ== --osd-uuid 4a7bb2d2-77b5-43cf-9a40-0b63b3f6e7f9 2026-02-25T16:37:09.665 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:09.664+0000 7f15a1655780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:09.667 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:09.665+0000 7f15a1655780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:09.668 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:09.666+0000 7f15a1655780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:09.668 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:09.666+0000 7f15a1655780 -1 bdev(0x26b0c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:09.668 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:09.666+0000 7f15a1655780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:10.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:10.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:10.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:10.130 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:10.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:10.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:10.230 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:10.231 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:10.231 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:10.231 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:10.231 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:10.244 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:10.242+0000 7fb13204f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:10.245 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:10.243+0000 7fb13204f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:10.246 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:10.244+0000 7fb13204f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:10.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:10.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:10.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:10.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:10.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:10.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:10.327 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:10.328 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:10.328 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:10.328 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:10.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:10.446 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:10.444+0000 7fb13204f780 -1 Falling back to public interface 2026-02-25T16:37:10.746 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:10.744+0000 7fb13204f780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:11.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:11.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:11.422 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:11.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:11.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:11.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:11.516 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1903137382,v1:127.0.0.1:6801/1903137382] [v2:127.0.0.1:6802/1903137382,v1:127.0.0.1:6803/1903137382] exists,up 4a7bb2d2-77b5-43cf-9a40-0b63b3f6e7f9 2026-02-25T16:37:11.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:11.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:11.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:11.518 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2026-02-25T16:37:11.518 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:11.519 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:11.519 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:11.519 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.519 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.519 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:11.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2026-02-25T16:37:11.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:11.561 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:11.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:11.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:11.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:11.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:11.675 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:11.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:11.677 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:11.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:11.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:11.678 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:11.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:11.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:11.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:11.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:11.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:11.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:11.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:11.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:11.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:11.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:11.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:11.688 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:11.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:11.689 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:11.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:11.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:11.690 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:11.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:11.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:11.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:11.691 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:11.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:11.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:11.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:11.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:11.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:11.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:11.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:11.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:11.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:11.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:11.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2026-02-25T16:37:11.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:11.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:11.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:11.717 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:11.717 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.717 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:11.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2026-02-25T16:37:11.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:11.742 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:11.743 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:11.743 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.743 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.744 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:11.744 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:11.744 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:11.786 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:11.786 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:11.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:11.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:11.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:11.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:11.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:11.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:11.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:11.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:11.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:11.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:11.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:11.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:11.929 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:11.929 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.929 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:11.930 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:11.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:11.945 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:11.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:11.984 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:11.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2026-02-25T16:37:11.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:11.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:11.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:11.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:11.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:11.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:11.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=af2576ac-e31b-436d-8e44-a168ffcc4439 2026-02-25T16:37:11.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 af2576ac-e31b-436d-8e44-a168ffcc4439' 2026-02-25T16:37:12.000 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 af2576ac-e31b-436d-8e44-a168ffcc4439 2026-02-25T16:37:12.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:12.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4JZ9pXWsOABAAau+XZQp0TNifF77jYGuxbA== 2026-02-25T16:37:12.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4JZ9pXWsOABAAau+XZQp0TNifF77jYGuxbA=="}' 2026-02-25T16:37:12.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new af2576ac-e31b-436d-8e44-a168ffcc4439 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:12.104 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:12.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:12.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQA4JZ9pXWsOABAAau+XZQp0TNifF77jYGuxbA== --osd-uuid af2576ac-e31b-436d-8e44-a168ffcc4439 2026-02-25T16:37:12.126 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.124+0000 7f2ab1e35780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:12.127 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.126+0000 7f2ab1e35780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:12.128 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.126+0000 7f2ab1e35780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:12.129 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.127+0000 7f2ab1e35780 -1 bdev(0x3de4c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:12.129 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.127+0000 7f2ab1e35780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:12.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:12.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:12.563 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:12.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:12.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:12.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:12.664 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:12.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2026-02-25T16:37:12.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:12.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:12.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:12.678 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.676+0000 7f17e75d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:12.679 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.678+0000 7f17e75d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:12.680 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.678+0000 7f17e75d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:12.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:12.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:12.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:12.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:12.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:12.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:12.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:12.760 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:12.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:12.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:12.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:12.881 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:12.880+0000 7f17e75d8780 -1 Falling back to public interface 2026-02-25T16:37:13.230 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:13.228+0000 7f17e75d8780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:13.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:13.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:13.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:13.855 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:13.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:13.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:13.952 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3631540627,v1:127.0.0.1:6801/3631540627] [v2:127.0.0.1:6802/3631540627,v1:127.0.0.1:6803/3631540627] exists,up af2576ac-e31b-436d-8e44-a168ffcc4439 2026-02-25T16:37:13.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:13.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:13.996 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:13.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:14.000 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:11.731+0000 7f028041fc00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T16:37:14.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:14.001 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:12.880+0000 7f17e75d8780 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:14.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:14.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:14.110 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:14.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:14.111 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:14.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:14.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:14.113 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:14.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:14.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:14.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:14.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:14.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:14.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:14.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:14.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:14.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:14.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:14.123 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:14.124 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:14.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:14.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:14.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:14.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:14.124 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:14.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:14.125 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:14.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:14.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:14.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:14.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:14.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:14.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:14.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:14.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:14.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:14.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:14.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:14.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:14.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:14.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:14.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:14.130 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:14.131 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.131 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:14.132 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:14.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:14.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:14.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T16:37:14.150 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:14.150 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:14.150 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:14.150 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:14.150 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.150 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.150 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:14.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T16:37:14.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:14.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:14.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:14.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:14.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:14.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:14.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:14.218 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:14.218 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:14.218 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:14.218 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:14.218 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:14.219 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:14.219 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:14.219 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:14.220 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:14.220 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.220 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.220 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:14.220 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:14.220 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:14.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:14.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:14.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:14.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:14.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:14.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:14.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:14.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:14.357 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:14.357 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:14.357 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:14.357 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.357 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.357 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:14.357 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:14.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:14.373 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T16:37:14.373 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:14.373 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:14.373 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:14.373 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.373 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.374 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:14.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:14.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:14.411 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:14.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T16:37:14.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:14.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:14.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a844bf8-afe0-44a3-9d82-03c79eb75042 2026-02-25T16:37:14.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3a844bf8-afe0-44a3-9d82-03c79eb75042' 2026-02-25T16:37:14.422 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 3a844bf8-afe0-44a3-9d82-03c79eb75042 2026-02-25T16:37:14.422 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:14.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6JZ9pHl1sGRAAV+6qBY28Lahqv7AP6JCKfQ== 2026-02-25T16:37:14.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6JZ9pHl1sGRAAV+6qBY28Lahqv7AP6JCKfQ=="}' 2026-02-25T16:37:14.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a844bf8-afe0-44a3-9d82-03c79eb75042 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:14.530 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:14.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:14.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQA6JZ9pHl1sGRAAV+6qBY28Lahqv7AP6JCKfQ== --osd-uuid 3a844bf8-afe0-44a3-9d82-03c79eb75042 2026-02-25T16:37:14.551 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:14.550+0000 7f426122d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:14.552 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:14.551+0000 7f426122d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:14.553 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:14.552+0000 7f426122d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:14.554 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:14.552+0000 7f426122d780 -1 bdev(0x2d8ac00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:14.554 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:14.552+0000 7f426122d780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:14.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:14.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:14.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:14.967 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:14.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:15.070 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:15.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:15.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2026-02-25T16:37:15.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:15.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:15.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:15.084 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:15.082+0000 7f1e7202e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:15.085 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:15.083+0000 7f1e7202e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:15.086 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:15.084+0000 7f1e7202e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:15.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:15.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:15.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:15.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:15.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:15.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:15.169 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:15.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:15.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:15.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:15.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:15.280 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:15.279+0000 7f1e7202e780 -1 Falling back to public interface 2026-02-25T16:37:15.685 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:15.684+0000 7f1e7202e780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:16.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:16.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:16.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:16.265 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:16.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:16.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:16.358 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2978194432,v1:127.0.0.1:6801/2978194432] [v2:127.0.0.1:6802/2978194432,v1:127.0.0.1:6803/2978194432] exists,up 3a844bf8-afe0-44a3-9d82-03c79eb75042 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.359 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:16.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:16.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:16.405 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:16.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:16.410 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:14.163+0000 7fc42d46fc00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T16:37:16.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:16.411 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:15.279+0000 7f1e7202e780 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:16.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:16.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:16.519 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:16.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:16.520 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:16.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:16.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:16.521 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:16.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:16.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:16.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:16.522 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:16.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:16.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:16.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:16.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:16.527 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.527 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:16.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:16.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:16.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:16.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:16.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:16.531 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:16.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:16.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:16.532 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:16.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:16.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:16.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:16.533 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:16.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:16.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:16.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:16.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:16.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:16.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:16.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:16.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:16.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:16.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:16.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:16.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:16.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:16.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:16.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T16:37:16.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:16.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:16.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:16.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:16.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:16.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T16:37:16.557 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:16.557 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:16.557 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:16.558 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:16.558 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.558 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.558 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:16.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T16:37:16.580 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:16.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:16.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:16.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:16.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:16.581 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:16.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:16.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:16.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:16.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:16.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:16.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:16.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:16.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:16.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:16.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:16.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:16.760 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:16.760 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.760 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:16.761 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:16.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:16.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T16:37:16.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:16.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:16.778 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:16.778 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.778 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.778 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:16.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:16.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:16.815 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:16.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:16.823 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:16.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:16.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2026-02-25T16:37:16.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:16.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:16.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b26c4a45-d238-4fa8-88b2-06a4f8994d98 2026-02-25T16:37:16.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b26c4a45-d238-4fa8-88b2-06a4f8994d98' 2026-02-25T16:37:16.825 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 b26c4a45-d238-4fa8-88b2-06a4f8994d98 2026-02-25T16:37:16.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:16.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8JZ9plLZzMRAArwHvblF2PrG99uPQ2ciWow== 2026-02-25T16:37:16.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8JZ9plLZzMRAArwHvblF2PrG99uPQ2ciWow=="}' 2026-02-25T16:37:16.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b26c4a45-d238-4fa8-88b2-06a4f8994d98 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:16.931 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:16.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:16.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQA8JZ9plLZzMRAArwHvblF2PrG99uPQ2ciWow== --osd-uuid b26c4a45-d238-4fa8-88b2-06a4f8994d98 2026-02-25T16:37:16.953 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:16.951+0000 7f4d0aa6d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:16.954 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:16.953+0000 7f4d0aa6d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:16.955 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:16.954+0000 7f4d0aa6d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:16.955 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:16.954+0000 7f4d0aa6d780 -1 bdev(0x2956c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:16.956 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:16.954+0000 7f4d0aa6d780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:17.363 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:17.363 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:17.363 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:17.364 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:17.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:17.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:17.463 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:17.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2026-02-25T16:37:17.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:17.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:17.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:17.476 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:17.474+0000 7f6854dd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:17.478 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:17.476+0000 7f6854dd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:17.478 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:17.477+0000 7f6854dd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:17.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:17.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:17.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:17.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:17.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:17.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:17.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:17.559 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:17.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:17.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:17.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:17.682 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:17.681+0000 7f6854dd4780 -1 Falling back to public interface 2026-02-25T16:37:18.082 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:18.080+0000 7f6854dd4780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:18.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:18.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:18.654 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:18.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:18.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:18.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:18.746 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2356164162,v1:127.0.0.1:6801/2356164162] [v2:127.0.0.1:6802/2356164162,v1:127.0.0.1:6803/2356164162] exists,up b26c4a45-d238-4fa8-88b2-06a4f8994d98 2026-02-25T16:37:18.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:18.747 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:18.748 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:18.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:18.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:18.793 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:18.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:18.801 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:16.571+0000 7f92db638c00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T16:37:18.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:18.802 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:17.681+0000 7f6854dd4780 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:18.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:18.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:18.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:18.909 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:18.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:18.910 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:18.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:18.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:18.911 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:18.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:18.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:18.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:18.912 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:18.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:18.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:18.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:18.917 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:18.917 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:18.917 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:18.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:18.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:18.920 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:18.920 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:18.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:18.921 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:18.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:18.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:18.922 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:18.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:18.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:18.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:18.924 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:18.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:18.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:18.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:18.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:18.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:18.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:18.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:18.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:18.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:18.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:18.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:18.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:18.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:18.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:18.930 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:18.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2026-02-25T16:37:18.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:18.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:18.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:18.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:18.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:18.948 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:18.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:18.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2026-02-25T16:37:18.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:18.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:18.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:18.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:18.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:18.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:18.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:19.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:19.016 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:19.017 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:19.017 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:19.017 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:19.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:19.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:19.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:19.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:19.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:19.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:19.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:19.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:19.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:19.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:19.211 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:19.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2026-02-25T16:37:19.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:19.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:19.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:19.219 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:19.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b0d1b3a7-de11-438d-a387-1ff3dc5e2d36 2026-02-25T16:37:19.221 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 b0d1b3a7-de11-438d-a387-1ff3dc5e2d36 2026-02-25T16:37:19.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b0d1b3a7-de11-438d-a387-1ff3dc5e2d36' 2026-02-25T16:37:19.222 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:19.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA/JZ9pd9t9DRAAZXDEQkXnPXQOJ9MtbFOI8Q== 2026-02-25T16:37:19.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA/JZ9pd9t9DRAAZXDEQkXnPXQOJ9MtbFOI8Q=="}' 2026-02-25T16:37:19.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b0d1b3a7-de11-438d-a387-1ff3dc5e2d36 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:19.330 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:19.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:19.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQA/JZ9pd9t9DRAAZXDEQkXnPXQOJ9MtbFOI8Q== --osd-uuid b0d1b3a7-de11-438d-a387-1ff3dc5e2d36 2026-02-25T16:37:19.352 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.351+0000 7f50f05e3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:19.354 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.352+0000 7f50f05e3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:19.355 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.353+0000 7f50f05e3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:19.355 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.353+0000 7f50f05e3780 -1 bdev(0x37acc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:19.355 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.354+0000 7f50f05e3780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:19.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:19.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:19.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:19.786 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:19.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:19.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:19.886 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:19.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2026-02-25T16:37:19.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:19.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:19.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:19.901 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.899+0000 7fcf1ac14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:19.902 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.900+0000 7fcf1ac14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:19.903 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:19.901+0000 7fcf1ac14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:19.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:19.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:19.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:19.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:19.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:19.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:19.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:19.982 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:19.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:19.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:20.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:20.126 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:20.124+0000 7fcf1ac14780 -1 Falling back to public interface 2026-02-25T16:37:20.484 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:20.483+0000 7fcf1ac14780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:21.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:21.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:21.079 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:21.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:21.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:21.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:21.174 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1254563911,v1:127.0.0.1:6801/1254563911] [v2:127.0.0.1:6802/1254563911,v1:127.0.0.1:6803/1254563911] exists,up b0d1b3a7-de11-438d-a387-1ff3dc5e2d36 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:21.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:21.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:21.219 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:21.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:21.224 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:18.962+0000 7f561002fc00 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T16:37:21.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:21.225 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:20.125+0000 7fcf1ac14780 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:21.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:21.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:21.334 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:21.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:21.335 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:21.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:21.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:21.337 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:21.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:21.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:21.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:21.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:21.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:21.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:21.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:21.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:21.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:21.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:21.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:21.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:21.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:21.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:21.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:21.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:21.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:21.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:21.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:21.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:21.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:21.351 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:21.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:21.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.354 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:21.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:21.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:21.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:21.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:21.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.356 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:21.358 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:21.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:21.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2026-02-25T16:37:21.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:21.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:21.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:21.375 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:21.375 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.375 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:21.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2026-02-25T16:37:21.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:21.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:21.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:21.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:21.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:21.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:21.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:21.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:21.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:21.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:21.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:21.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:21.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:21.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:21.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:21.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:21.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:21.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:21.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:21.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:21.583 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:21.583 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.583 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:21.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:21.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:21.599 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T16:37:21.599 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:21.599 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:21.599 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:21.599 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.600 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.600 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:21.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:21.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:21.638 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:21.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:21.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:21.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2026-02-25T16:37:21.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:21.649 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:21.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=deac2c88-4204-44f6-be8d-93a275361eb8 2026-02-25T16:37:21.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 deac2c88-4204-44f6-be8d-93a275361eb8' 2026-02-25T16:37:21.649 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 deac2c88-4204-44f6-be8d-93a275361eb8 2026-02-25T16:37:21.649 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:21.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBJZ9pR8cEJxAAu8BRQWFIkRqsWZYpmDvhVQ== 2026-02-25T16:37:21.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBJZ9pR8cEJxAAu8BRQWFIkRqsWZYpmDvhVQ=="}' 2026-02-25T16:37:21.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new deac2c88-4204-44f6-be8d-93a275361eb8 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:21.753 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:21.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:21.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQBBJZ9pR8cEJxAAu8BRQWFIkRqsWZYpmDvhVQ== --osd-uuid deac2c88-4204-44f6-be8d-93a275361eb8 2026-02-25T16:37:21.776 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:21.774+0000 7fa59d20f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:21.777 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:21.776+0000 7fa59d20f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:21.778 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:21.776+0000 7fa59d20f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:21.779 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:21.777+0000 7fa59d20f780 -1 bdev(0x2a6cc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:21.779 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:21.777+0000 7fa59d20f780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:22.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:22.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:22.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:22.171 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:22.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:22.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:22.272 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:22.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2026-02-25T16:37:22.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:22.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:22.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:22.286 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:22.284+0000 7f2884f3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:22.287 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:22.286+0000 7f2884f3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:22.288 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:22.286+0000 7f2884f3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:22.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:22.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:22.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:22.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:22.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:22.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:22.368 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:22.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:22.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:22.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:22.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:22.493 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:22.492+0000 7f2884f3d780 -1 Falling back to public interface 2026-02-25T16:37:22.801 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:22.799+0000 7f2884f3d780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:23.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:23.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:23.465 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:23.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:23.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:23.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:23.557 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3354855213,v1:127.0.0.1:6801/3354855213] [v2:127.0.0.1:6802/3354855213,v1:127.0.0.1:6803/3354855213] exists,up deac2c88-4204-44f6-be8d-93a275361eb8 2026-02-25T16:37:23.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:23.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:23.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:23.604 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:23.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:23.609 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:21.387+0000 7ffb5461dc00 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T16:37:23.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:23.611 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:22.492+0000 7f2884f3d780 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T16:37:23.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:23.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:23.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:23.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:23.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:23.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:23.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:23.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:23.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:23.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:23.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:23.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:23.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:23.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:23.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:23.719 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:23.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:23.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:23.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:23.721 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:23.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:23.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:23.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:23.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:23.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:23.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:23.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:23.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code-plugins 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:23.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:23.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:23.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:23.731 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:23.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:23.731 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:23.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:23.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:23.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:23.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:23.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:23.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:23.733 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:23.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:23.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:23.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:23.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:23.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:23.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:23.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:23.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code-plugins 2026-02-25T16:37:23.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:23.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.38253 2026-02-25T16:37:23.739 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:23.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-25T16:37:23.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2026-02-25T16:37:23.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:23.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:23.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:23.757 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:23.758 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.758 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.758 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:23.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2026-02-25T16:37:23.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:23.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.782 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:23.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:23.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get fsid 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:23.825 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:23.826 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:23.826 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.826 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.826 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:23.826 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:23.826 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.38253/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:23.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-25T16:37:23.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:23.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:23.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:23.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:23.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-25T16:37:23.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:23.959 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:23.959 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:23.959 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:23.960 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:23.960 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.960 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:23.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:23.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:23.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-25T16:37:23.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:23.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:23.976 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:23.976 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:23.977 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:23.977 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-mon.a.asok 2026-02-25T16:37:23.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:23.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-mon.a.asok log flush 2026-02-25T16:37:24.014 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:24.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:24.020 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:24.020 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2026-02-25T16:37:24.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-25T16:37:24.022 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:24.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=38f686e7-17e9-416a-a6b3-93e229d1c9d4 2026-02-25T16:37:24.022 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 38f686e7-17e9-416a-a6b3-93e229d1c9d4 2026-02-25T16:37:24.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 38f686e7-17e9-416a-a6b3-93e229d1c9d4' 2026-02-25T16:37:24.022 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:24.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEJZ9pcv3IARAAkl8e3S+XTFhuZPI7veAl1Q== 2026-02-25T16:37:24.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEJZ9pcv3IARAAkl8e3S+XTFhuZPI7veAl1Q=="}' 2026-02-25T16:37:24.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 38f686e7-17e9-416a-a6b3-93e229d1c9d4 -i td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:24.134 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:24.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-25T16:37:24.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQBEJZ9pcv3IARAAkl8e3S+XTFhuZPI7veAl1Q== --osd-uuid 38f686e7-17e9-416a-a6b3-93e229d1c9d4 2026-02-25T16:37:24.156 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.154+0000 7f22e926f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:24.157 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.156+0000 7f22e926f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:24.158 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.157+0000 7f22e926f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:24.159 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.157+0000 7f22e926f780 -1 bdev(0x2b1ac00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:24.159 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.157+0000 7f22e926f780 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-25T16:37:24.586 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-25T16:37:24.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:24.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:24.588 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:24.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:24.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:24.688 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:24.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=a3e34d4a-cd27-4fa3-a1f6-14b5b3bb827a --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.38253/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2026-02-25T16:37:24.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:24.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:24.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:24.701 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.699+0000 7f697a439780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:24.702 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.700+0000 7f697a439780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:24.703 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.701+0000 7f697a439780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:24.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:24.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:24.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:24.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:24.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:24.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:24.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:24.788 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:24.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:24.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:24.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:24.934 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:24.932+0000 7f697a439780 -1 Falling back to public interface 2026-02-25T16:37:25.267 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:25.265+0000 7f697a439780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:25.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:25.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:25.881 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:25.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:25.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:25.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:25.978 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/370948526,v1:127.0.0.1:6801/370948526] [v2:127.0.0.1:6802/370948526,v1:127.0.0.1:6803/370948526] exists,up 38f686e7-17e9-416a-a6b3-93e229d1c9d4 2026-02-25T16:37:25.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:25.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:25.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:25.979 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.38253/ceph-osd.0.asok 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-25T16:37:25.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.38253/ceph-osd.0.asok log flush 2026-02-25T16:37:26.024 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:26.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-25T16:37:26.029 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:23.770+0000 7fc742848c00 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T16:37:26.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:24.933+0000 7f697a439780 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:26.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:26.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:26.139 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:26.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:26.140 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:26.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:26.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:26.142 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:26.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:26.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:26.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:26.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:26.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:26.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:26.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:26.149 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:26.149 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.149 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:26.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/test-erasure-code-plugins 0 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code-plugins 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:26.151 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:26.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:26.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:26.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:26.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:26.153 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:26.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:26.153 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:26.155 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:26.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:26.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:26.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:26.156 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:26.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:26.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T16:37:26.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code-plugins 2026-02-25T16:37:26.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:26.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.38253 2026-02-25T16:37:26.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.38253 2026-02-25T16:37:26.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:26.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:26.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T16:37:26.162 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T16:37:26.162 DEBUG:teuthology.orchestra.run.trial129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T16:37:26.227 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2026-02-25T16:37:26.228 DEBUG:teuthology.orchestra.run.trial129:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2026-02-25T16:37:26.290 INFO:tasks.workunit.client.0.trial129.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/test-erasure-code 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2026-02-25T16:37:26.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2026-02-25T16:37:26.294 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2026-02-25T16:37:26.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none ' 2026-02-25T16:37:26.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:26.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-code 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-code 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:37:26.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:37:26.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:37:26.297 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:37:26.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:37:26.297 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:37:26.298 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:37:26.298 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:37:26.298 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:37:26.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:26.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:37:26.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:37:26.299 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:37:26.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:37:26.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:37:26.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2026-02-25T16:37:26.302 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:37:26.302 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.302 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.68264 2026-02-25T16:37:26.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:37:26.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:37:26.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-code 2026-02-25T16:37:26.304 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:37:26.304 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.304 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.68264 2026-02-25T16:37:26.305 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:37:26.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T16:37:26.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T16:37:26.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:37:26.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2026-02-25T16:37:26.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2026-02-25T16:37:26.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-code 2026-02-25T16:37:26.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:37:26.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:37:26.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:37:26.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-code/a 2026-02-25T16:37:26.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2026-02-25T16:37:26.324 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:37:26.324 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:26.324 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:26.324 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:26.325 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.325 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.325 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:26.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:37:26.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:37:26.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.350 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.68264/ceph-mon.a.asok 2026-02-25T16:37:26.351 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:26.351 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.68264/ceph-mon.a.asok config get fsid 2026-02-25T16:37:26.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:37:26.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:37:26.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:37:26.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:37:26.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:37:26.393 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:37:26.393 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:26.393 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:26.393 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:26.393 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.393 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.393 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.68264/ceph-mon.a.asok 2026-02-25T16:37:26.394 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:37:26.394 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.68264/ceph-mon.a.asok config get mon_host 2026-02-25T16:37:26.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2026-02-25T16:37:26.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-code 2026-02-25T16:37:26.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:37:26.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:37:26.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:37:26.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-code/x 2026-02-25T16:37:26.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:26.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:26.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.68264/ceph-mon.a.asok 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2026-02-25T16:37:26.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.68264/ceph-mon.a.asok log flush 2026-02-25T16:37:26.585 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:37:26.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2026-02-25T16:37:26.589 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:26.339+0000 7f4e9a411c00 0 load: jerasure load: lrc 2026-02-25T16:37:26.590 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/0 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:26.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:26.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/0 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8c7aec17-423f-40e7-bc73-2f20883fd99e 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8c7aec17-423f-40e7-bc73-2f20883fd99e' 2026-02-25T16:37:26.593 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 8c7aec17-423f-40e7-bc73-2f20883fd99e 2026-02-25T16:37:26.594 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:26.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBGJZ9pqsThIxAAxgx2s8Awp/PKk8GA96oG5A== 2026-02-25T16:37:26.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBGJZ9pqsThIxAAxgx2s8Awp/PKk8GA96oG5A=="}' 2026-02-25T16:37:26.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8c7aec17-423f-40e7-bc73-2f20883fd99e -i td/test-erasure-code/0/new.json 2026-02-25T16:37:26.705 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:26.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/0/new.json 2026-02-25T16:37:26.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGJZ9pqsThIxAAxgx2s8Awp/PKk8GA96oG5A== --osd-uuid 8c7aec17-423f-40e7-bc73-2f20883fd99e 2026-02-25T16:37:26.727 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:26.726+0000 7fbf2e211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:26.729 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:26.727+0000 7fbf2e211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:26.730 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:26.728+0000 7fbf2e211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:26.730 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:26.728+0000 7fbf2e211780 -1 bdev(0x3f8ac00 td/test-erasure-code/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:26.730 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:26.729+0000 7fbf2e211780 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2026-02-25T16:37:27.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/0/keyring 2026-02-25T16:37:27.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:27.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:37:27.135 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:37:27.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:27.235 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:37:27.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:37:27.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:27.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:27.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:27.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:27.250 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:27.248+0000 7fc386a3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:27.251 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:27.250+0000 7fc386a3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:27.252 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:27.251+0000 7fc386a3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:27.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:37:27.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:27.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:37:27.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:27.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:27.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:27.334 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:27.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:27.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:27.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:27.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:27.466 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:27.465+0000 7fc386a3e780 -1 Falling back to public interface 2026-02-25T16:37:27.895 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:27.893+0000 7fc386a3e780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:37:28.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:28.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:28.431 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:28.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:28.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:28.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:37:28.529 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/480467678,v1:127.0.0.1:6801/480467678] [v2:127.0.0.1:6802/480467678,v1:127.0.0.1:6803/480467678] exists,up 8c7aec17-423f-40e7-bc73-2f20883fd99e 2026-02-25T16:37:28.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:28.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:28.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:28.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:28.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2026-02-25T16:37:28.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:28.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/1 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:28.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:28.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/1 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c348cb41-65b3-4bc6-ad92-77c73c5ab4e7 2026-02-25T16:37:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c348cb41-65b3-4bc6-ad92-77c73c5ab4e7' 2026-02-25T16:37:28.538 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 c348cb41-65b3-4bc6-ad92-77c73c5ab4e7 2026-02-25T16:37:28.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:28.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBIJZ9pSulAIBAA6qClE07//w0BJySWouYyww== 2026-02-25T16:37:28.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBIJZ9pSulAIBAA6qClE07//w0BJySWouYyww=="}' 2026-02-25T16:37:28.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c348cb41-65b3-4bc6-ad92-77c73c5ab4e7 -i td/test-erasure-code/1/new.json 2026-02-25T16:37:28.641 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:28.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/1/new.json 2026-02-25T16:37:28.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIJZ9pSulAIBAA6qClE07//w0BJySWouYyww== --osd-uuid c348cb41-65b3-4bc6-ad92-77c73c5ab4e7 2026-02-25T16:37:28.663 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:28.661+0000 7f738ee56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:28.664 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:28.663+0000 7f738ee56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:28.665 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:28.663+0000 7f738ee56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:28.666 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:28.664+0000 7f738ee56780 -1 bdev(0x424ac00 td/test-erasure-code/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:28.666 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:28.664+0000 7f738ee56780 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2026-02-25T16:37:29.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/1/keyring 2026-02-25T16:37:29.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:29.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:37:29.028 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:37:29.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:29.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:37:29.343 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:37:29.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:29.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:29.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:29.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:29.357 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:29.355+0000 7f981121f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:29.358 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:29.357+0000 7f981121f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:29.359 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:29.357+0000 7f981121f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:29.520 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:29.519+0000 7f981121f780 -1 Falling back to public interface 2026-02-25T16:37:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:37:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:37:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:29.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:29.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:29.570 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:29.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:29.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:37:29.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:29.887 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:29.886+0000 7f981121f780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:37:30.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:30.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:30.795 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:30.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:30.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:30.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:37:31.021 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1068323132,v1:127.0.0.1:6811/1068323132] [v2:127.0.0.1:6812/1068323132,v1:127.0.0.1:6813/1068323132] exists,up c348cb41-65b3-4bc6-ad92-77c73c5ab4e7 2026-02-25T16:37:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2026-02-25T16:37:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/2 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:31.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:31.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/2 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=780d0c33-c8dc-400c-a709-99f5cc434b42 2026-02-25T16:37:31.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 780d0c33-c8dc-400c-a709-99f5cc434b42' 2026-02-25T16:37:31.029 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 780d0c33-c8dc-400c-a709-99f5cc434b42 2026-02-25T16:37:31.029 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:31.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBLJZ9p5zb5ARAAsVFzYQs0WLz1NeEPMUDB5Q== 2026-02-25T16:37:31.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBLJZ9p5zb5ARAAsVFzYQs0WLz1NeEPMUDB5Q=="}' 2026-02-25T16:37:31.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 780d0c33-c8dc-400c-a709-99f5cc434b42 -i td/test-erasure-code/2/new.json 2026-02-25T16:37:31.261 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:37:31.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/2/new.json 2026-02-25T16:37:31.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLJZ9p5zb5ARAAsVFzYQs0WLz1NeEPMUDB5Q== --osd-uuid 780d0c33-c8dc-400c-a709-99f5cc434b42 2026-02-25T16:37:31.283 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.281+0000 7f2cd4c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:31.285 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.283+0000 7f2cd4c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:31.285 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.284+0000 7f2cd4c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:31.286 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.284+0000 7f2cd4c32780 -1 bdev(0x3f54c00 td/test-erasure-code/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:31.286 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.284+0000 7f2cd4c32780 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2026-02-25T16:37:31.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/2/keyring 2026-02-25T16:37:31.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:31.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:37:31.644 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:37:31.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:31.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:37:31.963 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:37:31.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:31.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:31.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:31.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:31.978 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.976+0000 7f3adf411780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:31.979 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.977+0000 7f3adf411780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:31.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:31.978+0000 7f3adf411780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:32.145 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:32.143+0000 7f3adf411780 -1 Falling back to public interface 2026-02-25T16:37:32.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:37:32.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:32.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:37:32.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:32.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:32.190 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:32.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:32.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:32.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:32.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:37:32.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:32.507 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:32.505+0000 7f3adf411780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:37:33.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:33.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:33.414 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:33.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:33.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:33.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:37:33.639 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1958299671,v1:127.0.0.1:6819/1958299671] [v2:127.0.0.1:6820/1958299671,v1:127.0.0.1:6821/1958299671] exists,up 780d0c33-c8dc-400c-a709-99f5cc434b42 2026-02-25T16:37:33.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:33.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:33.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:33.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/3 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:33.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:33.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/3 2026-02-25T16:37:33.646 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:33.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=64c9513a-0f39-46a2-9911-a4eb3a0d003f 2026-02-25T16:37:33.647 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 64c9513a-0f39-46a2-9911-a4eb3a0d003f 2026-02-25T16:37:33.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 64c9513a-0f39-46a2-9911-a4eb3a0d003f' 2026-02-25T16:37:33.648 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:33.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNJZ9pPHbQJhAA52B9YbZx/cpYIFicmfoV8w== 2026-02-25T16:37:33.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNJZ9pPHbQJhAA52B9YbZx/cpYIFicmfoV8w=="}' 2026-02-25T16:37:33.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 64c9513a-0f39-46a2-9911-a4eb3a0d003f -i td/test-erasure-code/3/new.json 2026-02-25T16:37:33.884 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:37:33.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/3/new.json 2026-02-25T16:37:33.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNJZ9pPHbQJhAA52B9YbZx/cpYIFicmfoV8w== --osd-uuid 64c9513a-0f39-46a2-9911-a4eb3a0d003f 2026-02-25T16:37:33.907 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:33.905+0000 7f1e14c30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:33.908 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:33.907+0000 7f1e14c30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:33.909 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:33.908+0000 7f1e14c30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:33.910 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:33.908+0000 7f1e14c30780 -1 bdev(0x422cc00 td/test-erasure-code/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:33.910 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:33.908+0000 7f1e14c30780 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2026-02-25T16:37:34.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/3/keyring 2026-02-25T16:37:34.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:34.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:37:34.247 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:37:34.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:34.565 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:37:34.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:37:34.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:34.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:34.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:34.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:34.579 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:34.578+0000 7f3d1260c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:34.580 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:34.579+0000 7f3d1260c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:34.581 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:34.580+0000 7f3d1260c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:34.735 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:34.732+0000 7f3d1260c780 -1 Falling back to public interface 2026-02-25T16:37:34.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:37:34.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:34.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:37:35.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:35.123 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:35.121+0000 7f3d1260c780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:37:36.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:36.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:36.016 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:36.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:36.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:36.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:37:36.250 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1904641750,v1:127.0.0.1:6827/1904641750] [v2:127.0.0.1:6828/1904641750,v1:127.0.0.1:6829/1904641750] exists,up 64c9513a-0f39-46a2-9911-a4eb3a0d003f 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/4 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:36.251 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:36.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/4 2026-02-25T16:37:36.253 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:36.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6efe03dc-cdbb-4589-b88f-3f97911622f6 2026-02-25T16:37:36.254 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 6efe03dc-cdbb-4589-b88f-3f97911622f6 2026-02-25T16:37:36.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 6efe03dc-cdbb-4589-b88f-3f97911622f6' 2026-02-25T16:37:36.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:36.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBQJZ9pejijDxAAYQ+QctChr37wJnSgw52aYw== 2026-02-25T16:37:36.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBQJZ9pejijDxAAYQ+QctChr37wJnSgw52aYw=="}' 2026-02-25T16:37:36.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6efe03dc-cdbb-4589-b88f-3f97911622f6 -i td/test-erasure-code/4/new.json 2026-02-25T16:37:36.491 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:37:36.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/4/new.json 2026-02-25T16:37:36.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQJZ9pejijDxAAYQ+QctChr37wJnSgw52aYw== --osd-uuid 6efe03dc-cdbb-4589-b88f-3f97911622f6 2026-02-25T16:37:36.514 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:36.512+0000 7feddf268780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:36.515 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:36.513+0000 7feddf268780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:36.516 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:36.514+0000 7feddf268780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:36.516 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:36.515+0000 7feddf268780 -1 bdev(0x2d42c00 td/test-erasure-code/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:36.517 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:36.515+0000 7feddf268780 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2026-02-25T16:37:36.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/4/keyring 2026-02-25T16:37:36.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:36.860 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:37:36.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:37:36.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:37.177 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:37:37.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:37:37.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:37.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:37.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:37.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:37.191 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:37.190+0000 7fcedca3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:37.193 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:37.191+0000 7fcedca3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:37.193 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:37.192+0000 7fcedca3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:37.322 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:37.321+0000 7fcedca3e780 -1 Falling back to public interface 2026-02-25T16:37:37.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:37:37.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:37.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:37:37.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:37.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:37.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:37.409 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:37.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:37.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:37.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:37:37.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:37.648 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:37.647+0000 7fcedca3e780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:37:38.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:38.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:38.633 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:38.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:38.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:38.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:37:38.859 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/577290875,v1:127.0.0.1:6835/577290875] [v2:127.0.0.1:6836/577290875,v1:127.0.0.1:6837/577290875] exists,up 6efe03dc-cdbb-4589-b88f-3f97911622f6 2026-02-25T16:37:38.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:38.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:38.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:38.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:38.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2026-02-25T16:37:38.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/5 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:38.860 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:38.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/5 2026-02-25T16:37:38.862 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:38.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=46ee556a-e880-40db-ae25-113d5233f602 2026-02-25T16:37:38.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 46ee556a-e880-40db-ae25-113d5233f602' 2026-02-25T16:37:38.862 INFO:tasks.workunit.client.0.trial129.stdout:add osd5 46ee556a-e880-40db-ae25-113d5233f602 2026-02-25T16:37:38.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:38.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBSJZ9p6JryMxAAuarIM0nYwoH8NJpauEVgUw== 2026-02-25T16:37:38.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBSJZ9p6JryMxAAuarIM0nYwoH8NJpauEVgUw=="}' 2026-02-25T16:37:38.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 46ee556a-e880-40db-ae25-113d5233f602 -i td/test-erasure-code/5/new.json 2026-02-25T16:37:39.100 INFO:tasks.workunit.client.0.trial129.stdout:5 2026-02-25T16:37:39.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/5/new.json 2026-02-25T16:37:39.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSJZ9p6JryMxAAuarIM0nYwoH8NJpauEVgUw== --osd-uuid 46ee556a-e880-40db-ae25-113d5233f602 2026-02-25T16:37:39.124 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.122+0000 7fd681246780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:39.125 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.123+0000 7fd681246780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:39.126 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.124+0000 7fd681246780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:39.126 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.125+0000 7fd681246780 -1 bdev(0x26fac00 td/test-erasure-code/5/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:39.126 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.125+0000 7fd681246780 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2026-02-25T16:37:39.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/5/keyring 2026-02-25T16:37:39.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:39.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T16:37:39.476 INFO:tasks.workunit.client.0.trial129.stdout:adding osd5 key to auth repository 2026-02-25T16:37:39.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:39.794 INFO:tasks.workunit.client.0.trial129.stdout:start osd.5 2026-02-25T16:37:39.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T16:37:39.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:39.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:39.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:39.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:39.808 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.806+0000 7feb8bc0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:39.810 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.808+0000 7feb8bc0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:39.810 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.809+0000 7feb8bc0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:39.979 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:39.978+0000 7feb8bc0c780 -1 Falling back to public interface 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:40.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:37:40.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:40.325 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:40.323+0000 7feb8bc0c780 -1 osd.5 0 log_to_monitors true 2026-02-25T16:37:41.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:41.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:41.253 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:41.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:41.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:41.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:37:41.481 INFO:tasks.workunit.client.0.trial129.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/791028852,v1:127.0.0.1:6843/791028852] [v2:127.0.0.1:6844/791028852,v1:127.0.0.1:6845/791028852] exists,up 46ee556a-e880-40db-ae25-113d5233f602 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T16:37:41.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/6 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:41.487 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:41.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/6 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a2a6f51-6e33-4f14-8ee5-ef90410a7cbc 2026-02-25T16:37:41.489 INFO:tasks.workunit.client.0.trial129.stdout:add osd6 4a2a6f51-6e33-4f14-8ee5-ef90410a7cbc 2026-02-25T16:37:41.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 4a2a6f51-6e33-4f14-8ee5-ef90410a7cbc' 2026-02-25T16:37:41.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:41.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBVJZ9pGmVnHRAANOVHPfjtaNzCr4Xhz8v75g== 2026-02-25T16:37:41.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBVJZ9pGmVnHRAANOVHPfjtaNzCr4Xhz8v75g=="}' 2026-02-25T16:37:41.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a2a6f51-6e33-4f14-8ee5-ef90410a7cbc -i td/test-erasure-code/6/new.json 2026-02-25T16:37:41.724 INFO:tasks.workunit.client.0.trial129.stdout:6 2026-02-25T16:37:41.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/6/new.json 2026-02-25T16:37:41.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVJZ9pGmVnHRAANOVHPfjtaNzCr4Xhz8v75g== --osd-uuid 4a2a6f51-6e33-4f14-8ee5-ef90410a7cbc 2026-02-25T16:37:41.747 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:41.746+0000 7f3df6c15780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:41.749 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:41.747+0000 7f3df6c15780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:41.749 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:41.748+0000 7f3df6c15780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:41.750 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:41.748+0000 7f3df6c15780 -1 bdev(0x2b1ac00 td/test-erasure-code/6/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:41.750 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:41.748+0000 7f3df6c15780 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2026-02-25T16:37:42.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/6/keyring 2026-02-25T16:37:42.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:42.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T16:37:42.081 INFO:tasks.workunit.client.0.trial129.stdout:adding osd6 key to auth repository 2026-02-25T16:37:42.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:42.403 INFO:tasks.workunit.client.0.trial129.stdout:start osd.6 2026-02-25T16:37:42.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T16:37:42.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:42.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:42.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:42.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:42.417 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:42.416+0000 7f4065e33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:42.419 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:42.417+0000 7f4065e33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:42.420 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:42.418+0000 7f4065e33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:42.562 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:42.560+0000 7f4065e33780 -1 Falling back to public interface 2026-02-25T16:37:42.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T16:37:42.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:42.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T16:37:42.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:42.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:42.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:42.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:42.626 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:42.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:42.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:37:42.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:42.936 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:42.934+0000 7f4065e33780 -1 osd.6 0 log_to_monitors true 2026-02-25T16:37:43.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:43.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:43.850 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:43.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:43.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:43.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:37:44.075 INFO:tasks.workunit.client.0.trial129.stdout:osd.6 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2797959233,v1:127.0.0.1:6851/2797959233] [v2:127.0.0.1:6852/2797959233,v1:127.0.0.1:6853/2797959233] exists,up 4a2a6f51-6e33-4f14-8ee5-ef90410a7cbc 2026-02-25T16:37:44.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:44.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:44.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:44.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:44.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2026-02-25T16:37:44.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/7 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2026-02-25T16:37:44.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:44.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/7 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:44.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=baec7126-49b9-445d-9970-8c3bcdddc7b3 2026-02-25T16:37:44.083 INFO:tasks.workunit.client.0.trial129.stdout:add osd7 baec7126-49b9-445d-9970-8c3bcdddc7b3 2026-02-25T16:37:44.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 baec7126-49b9-445d-9970-8c3bcdddc7b3' 2026-02-25T16:37:44.083 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:44.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBYJZ9paYY/BRAAGSzSK2f4XqYV0aGy+kA/tA== 2026-02-25T16:37:44.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBYJZ9paYY/BRAAGSzSK2f4XqYV0aGy+kA/tA=="}' 2026-02-25T16:37:44.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new baec7126-49b9-445d-9970-8c3bcdddc7b3 -i td/test-erasure-code/7/new.json 2026-02-25T16:37:44.318 INFO:tasks.workunit.client.0.trial129.stdout:7 2026-02-25T16:37:44.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/7/new.json 2026-02-25T16:37:44.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYJZ9paYY/BRAAGSzSK2f4XqYV0aGy+kA/tA== --osd-uuid baec7126-49b9-445d-9970-8c3bcdddc7b3 2026-02-25T16:37:44.341 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:44.339+0000 7f1ad9ffd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:44.342 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:44.341+0000 7f1ad9ffd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:44.343 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:44.342+0000 7f1ad9ffd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:44.343 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:44.342+0000 7f1ad9ffd780 -1 bdev(0x4322c00 td/test-erasure-code/7/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:44.344 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:44.342+0000 7f1ad9ffd780 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2026-02-25T16:37:44.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/7/keyring 2026-02-25T16:37:44.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:44.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-25T16:37:44.722 INFO:tasks.workunit.client.0.trial129.stdout:adding osd7 key to auth repository 2026-02-25T16:37:44.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:45.040 INFO:tasks.workunit.client.0.trial129.stdout:start osd.7 2026-02-25T16:37:45.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-25T16:37:45.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:45.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:45.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:45.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:45.054 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:45.053+0000 7f407a42c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:45.055 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:45.054+0000 7f407a42c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:45.056 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:45.055+0000 7f407a42c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:45.233 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:45.231+0000 7f407a42c780 -1 Falling back to public interface 2026-02-25T16:37:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-25T16:37:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-25T16:37:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:45.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:45.266 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:45.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:45.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T16:37:45.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:45.602 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:45.600+0000 7f407a42c780 -1 osd.7 0 log_to_monitors true 2026-02-25T16:37:46.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:46.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:46.490 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:46.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:46.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:46.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T16:37:46.713 INFO:tasks.workunit.client.0.trial129.stdout:osd.7 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4107790857,v1:127.0.0.1:6859/4107790857] [v2:127.0.0.1:6860/4107790857,v1:127.0.0.1:6861/4107790857] exists,up baec7126-49b9-445d-9970-8c3bcdddc7b3 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:46.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/8 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:46.715 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:46.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/8 2026-02-25T16:37:46.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:46.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4b60648d-9c29-4f4b-9689-d01bd5267608 2026-02-25T16:37:46.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 4b60648d-9c29-4f4b-9689-d01bd5267608' 2026-02-25T16:37:46.717 INFO:tasks.workunit.client.0.trial129.stdout:add osd8 4b60648d-9c29-4f4b-9689-d01bd5267608 2026-02-25T16:37:46.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:46.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBaJZ9pZ7w6KxAAIFlDGE+VWBpYBAGAkMPSQA== 2026-02-25T16:37:46.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBaJZ9pZ7w6KxAAIFlDGE+VWBpYBAGAkMPSQA=="}' 2026-02-25T16:37:46.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4b60648d-9c29-4f4b-9689-d01bd5267608 -i td/test-erasure-code/8/new.json 2026-02-25T16:37:46.955 INFO:tasks.workunit.client.0.trial129.stdout:8 2026-02-25T16:37:46.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/8/new.json 2026-02-25T16:37:46.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaJZ9pZ7w6KxAAIFlDGE+VWBpYBAGAkMPSQA== --osd-uuid 4b60648d-9c29-4f4b-9689-d01bd5267608 2026-02-25T16:37:46.978 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:46.976+0000 7f7061033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:46.979 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:46.978+0000 7f7061033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:46.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:46.978+0000 7f7061033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:46.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:46.979+0000 7f7061033780 -1 bdev(0x41b4c00 td/test-erasure-code/8/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:46.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:46.979+0000 7f7061033780 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2026-02-25T16:37:47.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/8/keyring 2026-02-25T16:37:47.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:47.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-25T16:37:47.344 INFO:tasks.workunit.client.0.trial129.stdout:adding osd8 key to auth repository 2026-02-25T16:37:47.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:47.659 INFO:tasks.workunit.client.0.trial129.stdout:start osd.8 2026-02-25T16:37:47.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-25T16:37:47.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:47.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:47.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:47.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:47.673 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:47.671+0000 7fa21b042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:47.674 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:47.673+0000 7fa21b042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:47.675 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:47.673+0000 7fa21b042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:47.835 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:47.834+0000 7fa21b042780 -1 Falling back to public interface 2026-02-25T16:37:47.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-25T16:37:47.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:47.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-25T16:37:47.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:47.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:47.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:47.892 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:47.893 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:47.893 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:47.893 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T16:37:48.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:48.190 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:48.189+0000 7fa21b042780 -1 osd.8 0 log_to_monitors true 2026-02-25T16:37:49.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:49.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:49.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:49.118 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:49.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:49.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T16:37:49.346 INFO:tasks.workunit.client.0.trial129.stdout:osd.8 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1272729222,v1:127.0.0.1:6867/1272729222] [v2:127.0.0.1:6868/1272729222,v1:127.0.0.1:6869/1272729222] exists,up 4b60648d-9c29-4f4b-9689-d01bd5267608 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/9 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:49.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:49.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:49.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:49.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:49.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:49.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/9 2026-02-25T16:37:49.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:49.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e2ab4810-4d41-4861-8151-2b72f14a40df 2026-02-25T16:37:49.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 e2ab4810-4d41-4861-8151-2b72f14a40df' 2026-02-25T16:37:49.350 INFO:tasks.workunit.client.0.trial129.stdout:add osd9 e2ab4810-4d41-4861-8151-2b72f14a40df 2026-02-25T16:37:49.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:49.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdJZ9pQ4tdFRAA88WHSZRRu5fZX1rstU7pMA== 2026-02-25T16:37:49.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdJZ9pQ4tdFRAA88WHSZRRu5fZX1rstU7pMA=="}' 2026-02-25T16:37:49.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e2ab4810-4d41-4861-8151-2b72f14a40df -i td/test-erasure-code/9/new.json 2026-02-25T16:37:49.588 INFO:tasks.workunit.client.0.trial129.stdout:9 2026-02-25T16:37:49.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/9/new.json 2026-02-25T16:37:49.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdJZ9pQ4tdFRAA88WHSZRRu5fZX1rstU7pMA== --osd-uuid e2ab4810-4d41-4861-8151-2b72f14a40df 2026-02-25T16:37:49.613 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:49.611+0000 7f191362e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:49.614 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:49.613+0000 7f191362e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:49.615 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:49.614+0000 7f191362e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:49.615 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:49.614+0000 7f191362e780 -1 bdev(0x3f22c00 td/test-erasure-code/9/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:49.615 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:49.614+0000 7f191362e780 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2026-02-25T16:37:49.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/9/keyring 2026-02-25T16:37:49.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:49.945 INFO:tasks.workunit.client.0.trial129.stdout:adding osd9 key to auth repository 2026-02-25T16:37:49.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-25T16:37:49.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:50.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-25T16:37:50.262 INFO:tasks.workunit.client.0.trial129.stdout:start osd.9 2026-02-25T16:37:50.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:50.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:50.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:50.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:50.275 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:50.273+0000 7f501c65e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:50.276 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:50.274+0000 7f501c65e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:50.277 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:50.275+0000 7f501c65e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:50.425 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:50.423+0000 7f501c65e780 -1 Falling back to public interface 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:50.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T16:37:50.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:50.786 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:50.784+0000 7f501c65e780 -1 osd.9 0 log_to_monitors true 2026-02-25T16:37:51.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:51.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:51.717 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:51.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:51.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:51.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T16:37:51.949 INFO:tasks.workunit.client.0.trial129.stdout:osd.9 up in weight 1 up_from 43 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2116702666,v1:127.0.0.1:6875/2116702666] [v2:127.0.0.1:6876/2116702666,v1:127.0.0.1:6877/2116702666] exists,up e2ab4810-4d41-4861-8151-2b72f14a40df 2026-02-25T16:37:51.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-code 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=10 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-code/10 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:37:51.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:37:51.955 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:37:51.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:37:51.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-code/10 2026-02-25T16:37:51.957 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:37:51.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a1a5e3c-fd5a-4df9-9161-c1fcdf99b2ce 2026-02-25T16:37:51.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd10 3a1a5e3c-fd5a-4df9-9161-c1fcdf99b2ce' 2026-02-25T16:37:51.957 INFO:tasks.workunit.client.0.trial129.stdout:add osd10 3a1a5e3c-fd5a-4df9-9161-c1fcdf99b2ce 2026-02-25T16:37:51.957 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:37:51.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBfJZ9p/BtUORAAN1iIkXBIvzQQFPcfztkzDQ== 2026-02-25T16:37:51.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBfJZ9p/BtUORAAN1iIkXBIvzQQFPcfztkzDQ=="}' 2026-02-25T16:37:51.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a1a5e3c-fd5a-4df9-9161-c1fcdf99b2ce -i td/test-erasure-code/10/new.json 2026-02-25T16:37:52.191 INFO:tasks.workunit.client.0.trial129.stdout:10 2026-02-25T16:37:52.198 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-code/10/new.json 2026-02-25T16:37:52.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 10 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfJZ9p/BtUORAAN1iIkXBIvzQQFPcfztkzDQ== --osd-uuid 3a1a5e3c-fd5a-4df9-9161-c1fcdf99b2ce 2026-02-25T16:37:52.214 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.212+0000 7fb56ec2b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:52.215 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.214+0000 7fb56ec2b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:52.216 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.214+0000 7fb56ec2b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:52.217 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.215+0000 7fb56ec2b780 -1 bdev(0x3264c00 td/test-erasure-code/10/block) open stat got: (1) Operation not permitted 2026-02-25T16:37:52.217 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.215+0000 7fb56ec2b780 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2026-02-25T16:37:52.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-code/10/keyring 2026-02-25T16:37:52.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:37:52.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd10 key to auth repository 2026-02-25T16:37:52.584 INFO:tasks.workunit.client.0.trial129.stdout:adding osd10 key to auth repository 2026-02-25T16:37:52.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:37:52.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.10 2026-02-25T16:37:52.903 INFO:tasks.workunit.client.0.trial129.stdout:start osd.10 2026-02-25T16:37:52.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 10 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:37:52.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:37:52.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:37:52.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:37:52.915 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.914+0000 7f4e6fe36780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:52.917 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.915+0000 7f4e6fe36780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:52.918 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:52.916+0000 7f4e6fe36780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:37:53.047 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:53.045+0000 7f4e6fe36780 -1 Falling back to public interface 2026-02-25T16:37:53.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 10 2026-02-25T16:37:53.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:37:53.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=10 2026-02-25T16:37:53.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:37:53.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:37:53.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:53.132 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:37:53.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:37:53.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:53.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-25T16:37:53.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:37:53.444 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:37:53.442+0000 7f4e6fe36780 -1 osd.10 0 log_to_monitors true 2026-02-25T16:37:54.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:37:54.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:37:54.361 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:37:54.362 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:37:54.362 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:37:54.362 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-25T16:37:54.584 INFO:tasks.workunit.client.0.trial129.stdout:osd.10 up in weight 1 up_from 47 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/3777349929,v1:127.0.0.1:6883/3777349929] [v2:127.0.0.1:6884/3777349929,v1:127.0.0.1:6885/3777349929] exists,up 3a1a5e3c-fd5a-4df9-9161-c1fcdf99b2ce 2026-02-25T16:37:54.585 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:37:54.585 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:37:54.585 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:37:54.585 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2026-02-25T16:37:54.586 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T16:37:54.808 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' does not exist 2026-02-25T16:37:54.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T16:37:54.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:37:55.087 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:37:55.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:37:56.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T16:37:56.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:37:56.341 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:37:56.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:37:56.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:37:56.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:37:56.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:37:56.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:37:56.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:37:56.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:37:56.652 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:37:56.652 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:37:56.652 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:37:56.652 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:56.653 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:37:56.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-25T16:37:56.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-25T16:37:56.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488' 2026-02-25T16:37:56.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:56.715 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:37:56.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705673 2026-02-25T16:37:56.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705673 2026-02-25T16:37:56.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673' 2026-02-25T16:37:56.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:56.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:37:56.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-25T16:37:56.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-25T16:37:56.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855' 2026-02-25T16:37:56.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:56.842 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:37:56.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411335 2026-02-25T16:37:56.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411335 2026-02-25T16:37:56.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335' 2026-02-25T16:37:56.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:56.908 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:37:56.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247814 2026-02-25T16:37:56.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247814 2026-02-25T16:37:56.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335 4-98784247814' 2026-02-25T16:37:56.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:56.970 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:37:57.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116998 2026-02-25T16:37:57.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116998 2026-02-25T16:37:57.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335 4-98784247814 5-115964116998' 2026-02-25T16:37:57.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:57.032 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:37:57.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986181 2026-02-25T16:37:57.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986181 2026-02-25T16:37:57.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335 4-98784247814 5-115964116998 6-133143986181' 2026-02-25T16:37:57.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:57.097 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:37:57.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855366 2026-02-25T16:37:57.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855366 2026-02-25T16:37:57.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335 4-98784247814 5-115964116998 6-133143986181 7-150323855366' 2026-02-25T16:37:57.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:57.162 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:37:57.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724549 2026-02-25T16:37:57.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724549 2026-02-25T16:37:57.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335 4-98784247814 5-115964116998 6-133143986181 7-150323855366 8-167503724549' 2026-02-25T16:37:57.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:57.225 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:37:57.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-25T16:37:57.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-25T16:37:57.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335 4-98784247814 5-115964116998 6-133143986181 7-150323855366 8-167503724549 9-184683593732' 2026-02-25T16:37:57.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:37:57.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:37:57.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462916 2026-02-25T16:37:57.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462916 2026-02-25T16:37:57.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-38654705673 2-55834574855 3-77309411335 4-98784247814 5-115964116998 6-133143986181 7-150323855366 8-167503724549 9-184683593732 10-201863462916' 2026-02-25T16:37:57.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:37:57.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836488 2026-02-25T16:37:57.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:37:57.354 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:37:57.355 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-25T16:37:57.355 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:37:57.356 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836488 2026-02-25T16:37:57.356 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-25T16:37:57.357 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 21474836488 2026-02-25T16:37:57.357 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:37:57.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836486 -lt 21474836488 2026-02-25T16:37:57.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:37:58.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:37:58.584 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:37:58.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-25T16:37:58.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:37:58.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705673 2026-02-25T16:37:58.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:37:58.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:37:58.812 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705673 2026-02-25T16:37:58.812 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:37:58.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705673 2026-02-25T16:37:58.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2026-02-25T16:37:58.813 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 38654705673 2026-02-25T16:37:58.814 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:37:59.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705673 -lt 38654705673 2026-02-25T16:37:59.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:37:59.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574855 2026-02-25T16:37:59.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:37:59.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:37:59.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574855 2026-02-25T16:37:59.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:37:59.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574855 2026-02-25T16:37:59.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2026-02-25T16:37:59.042 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 55834574855 2026-02-25T16:37:59.042 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:37:59.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-25T16:37:59.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:37:59.265 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-77309411335 2026-02-25T16:37:59.266 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:37:59.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:37:59.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411335 2026-02-25T16:37:59.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:37:59.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411335 2026-02-25T16:37:59.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 77309411335' 2026-02-25T16:37:59.269 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 77309411335 2026-02-25T16:37:59.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:37:59.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411335 -lt 77309411335 2026-02-25T16:37:59.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:37:59.493 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247814 2026-02-25T16:37:59.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:37:59.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:37:59.495 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247814 2026-02-25T16:37:59.495 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:37:59.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247814 2026-02-25T16:37:59.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247814' 2026-02-25T16:37:59.497 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 98784247814 2026-02-25T16:37:59.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:37:59.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247814 -lt 98784247814 2026-02-25T16:37:59.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:37:59.725 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-115964116998 2026-02-25T16:37:59.725 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:37:59.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:37:59.727 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964116998 2026-02-25T16:37:59.727 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:37:59.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116998 2026-02-25T16:37:59.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 115964116998' 2026-02-25T16:37:59.728 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 115964116998 2026-02-25T16:37:59.728 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:37:59.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116998 -lt 115964116998 2026-02-25T16:37:59.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:37:59.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986181 2026-02-25T16:37:59.954 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:37:59.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:37:59.955 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986181 2026-02-25T16:37:59.956 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:37:59.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986181 2026-02-25T16:37:59.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986181' 2026-02-25T16:37:59.957 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 133143986181 2026-02-25T16:37:59.957 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:38:00.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986181 -lt 133143986181 2026-02-25T16:38:00.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:00.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855366 2026-02-25T16:38:00.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:00.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:38:00.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855366 2026-02-25T16:38:00.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:00.182 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855366 2026-02-25T16:38:00.182 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855366' 2026-02-25T16:38:00.182 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.7 seq 150323855366 2026-02-25T16:38:00.183 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:38:00.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855366 -lt 150323855366 2026-02-25T16:38:00.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:00.409 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:00.409 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724549 2026-02-25T16:38:00.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:38:00.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724549 2026-02-25T16:38:00.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:00.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724549 2026-02-25T16:38:00.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724549' 2026-02-25T16:38:00.413 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.8 seq 167503724549 2026-02-25T16:38:00.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:38:00.637 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724550 -lt 167503724549 2026-02-25T16:38:00.637 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:00.637 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-184683593732 2026-02-25T16:38:00.637 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:00.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:38:00.639 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-184683593732 2026-02-25T16:38:00.639 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:00.640 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593732 2026-02-25T16:38:00.640 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 184683593732' 2026-02-25T16:38:00.640 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.9 seq 184683593732 2026-02-25T16:38:00.640 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:38:00.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-25T16:38:00.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:00.869 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462916 2026-02-25T16:38:00.870 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:00.870 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:38:00.871 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462916 2026-02-25T16:38:00.871 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:00.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462916 2026-02-25T16:38:00.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462916' 2026-02-25T16:38:00.872 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.10 seq 201863462916 2026-02-25T16:38:00.873 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:38:01.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462917 -lt 201863462916 2026-02-25T16:38:01.097 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:38:01.097 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:01.097 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:01.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T16:38:01.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:38:01.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:38:01.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:38:01.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:38:01.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:38:01.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:38:01.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:38:01.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T16:38:01.626 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:38:01.626 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:01.626 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:01.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T16:38:01.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:38:01.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:38:01.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2026-02-25T16:38:01.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:38:01.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:38:01.939 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:38:01.939 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:38:01.939 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:38:01.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.68264/ceph-osd.0.asok 2026-02-25T16:38:01.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2026-02-25T16:38:01.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.68264/ceph-osd.0.asok log flush 2026-02-25T16:38:01.983 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:38:01.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2026-02-25T16:38:01.988 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:37:27.467+0000 7fc386a3e780 0 load: jerasure load: lrc 2026-02-25T16:38:01.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2026-02-25T16:38:01.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2026-02-25T16:38:01.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2026-02-25T16:38:02.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2026-02-25T16:38:02.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2026-02-25T16:38:02.650 INFO:tasks.workunit.client.0.trial129.stderr:pool 'ecpool' already exists 2026-02-25T16:38:02.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:38:03.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:38:03.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:38:03.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:38:03.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:38:03.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:38:03.661 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:38:03.661 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:38:03.662 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:38:03.662 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:38:03.662 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:38:03.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:38:03.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:38:03.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:38:03.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:38:03.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:38:03.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:38:03.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:03.968 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:38:04.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836492 2026-02-25T16:38:04.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836492 2026-02-25T16:38:04.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492' 2026-02-25T16:38:04.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.033 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:38:04.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705676 2026-02-25T16:38:04.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705676 2026-02-25T16:38:04.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676' 2026-02-25T16:38:04.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.097 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:38:04.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574859 2026-02-25T16:38:04.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574859 2026-02-25T16:38:04.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859' 2026-02-25T16:38:04.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.161 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:38:04.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411338 2026-02-25T16:38:04.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411338 2026-02-25T16:38:04.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338' 2026-02-25T16:38:04.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.224 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:38:04.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247818 2026-02-25T16:38:04.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247818 2026-02-25T16:38:04.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338 4-98784247818' 2026-02-25T16:38:04.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:38:04.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117001 2026-02-25T16:38:04.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117001 2026-02-25T16:38:04.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338 4-98784247818 5-115964117001' 2026-02-25T16:38:04.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:38:04.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986185 2026-02-25T16:38:04.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986185 2026-02-25T16:38:04.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338 4-98784247818 5-115964117001 6-133143986185' 2026-02-25T16:38:04.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.418 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:38:04.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855369 2026-02-25T16:38:04.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855369 2026-02-25T16:38:04.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338 4-98784247818 5-115964117001 6-133143986185 7-150323855369' 2026-02-25T16:38:04.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.483 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:38:04.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724553 2026-02-25T16:38:04.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724553 2026-02-25T16:38:04.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338 4-98784247818 5-115964117001 6-133143986185 7-150323855369 8-167503724553' 2026-02-25T16:38:04.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.549 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:38:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593735 2026-02-25T16:38:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593735 2026-02-25T16:38:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338 4-98784247818 5-115964117001 6-133143986185 7-150323855369 8-167503724553 9-184683593735' 2026-02-25T16:38:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:04.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:38:04.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462920 2026-02-25T16:38:04.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462920 2026-02-25T16:38:04.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-55834574859 3-77309411338 4-98784247818 5-115964117001 6-133143986185 7-150323855369 8-167503724553 9-184683593735 10-201863462920' 2026-02-25T16:38:04.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:04.677 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836492 2026-02-25T16:38:04.677 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:04.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:38:04.678 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836492 2026-02-25T16:38:04.679 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:04.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836492 2026-02-25T16:38:04.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2026-02-25T16:38:04.680 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 21474836492 2026-02-25T16:38:04.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:04.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836492 -lt 21474836492 2026-02-25T16:38:04.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:04.905 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705676 2026-02-25T16:38:04.905 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:04.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:38:04.907 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705676 2026-02-25T16:38:04.907 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:04.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705676 2026-02-25T16:38:04.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2026-02-25T16:38:04.908 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 38654705676 2026-02-25T16:38:04.909 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:38:05.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705676 -lt 38654705676 2026-02-25T16:38:05.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:05.135 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574859 2026-02-25T16:38:05.135 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:05.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:38:05.136 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574859 2026-02-25T16:38:05.136 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:05.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574859 2026-02-25T16:38:05.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574859' 2026-02-25T16:38:05.138 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 55834574859 2026-02-25T16:38:05.138 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:38:05.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574859 -lt 55834574859 2026-02-25T16:38:05.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:05.361 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-77309411338 2026-02-25T16:38:05.362 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:05.362 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:38:05.363 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411338 2026-02-25T16:38:05.363 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:05.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411338 2026-02-25T16:38:05.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 77309411338' 2026-02-25T16:38:05.364 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 77309411338 2026-02-25T16:38:05.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:38:05.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411338 -lt 77309411338 2026-02-25T16:38:05.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:05.593 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247818 2026-02-25T16:38:05.593 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:05.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:38:05.594 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247818 2026-02-25T16:38:05.594 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:05.595 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247818 2026-02-25T16:38:05.595 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247818' 2026-02-25T16:38:05.595 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 98784247818 2026-02-25T16:38:05.596 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:38:05.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247818 -lt 98784247818 2026-02-25T16:38:05.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:05.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-115964117001 2026-02-25T16:38:05.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:05.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:38:05.819 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117001 2026-02-25T16:38:05.819 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:05.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117001 2026-02-25T16:38:05.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 115964117001' 2026-02-25T16:38:05.821 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 115964117001 2026-02-25T16:38:05.821 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:38:06.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117001 -lt 115964117001 2026-02-25T16:38:06.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:06.047 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986185 2026-02-25T16:38:06.047 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:06.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:38:06.048 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986185 2026-02-25T16:38:06.049 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:06.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986185 2026-02-25T16:38:06.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986185' 2026-02-25T16:38:06.050 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 133143986185 2026-02-25T16:38:06.050 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:38:06.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986185 -lt 133143986185 2026-02-25T16:38:06.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:06.276 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855369 2026-02-25T16:38:06.276 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:06.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:38:06.278 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855369 2026-02-25T16:38:06.278 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:06.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855369 2026-02-25T16:38:06.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855369' 2026-02-25T16:38:06.279 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.7 seq 150323855369 2026-02-25T16:38:06.279 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:38:06.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855369 -lt 150323855369 2026-02-25T16:38:06.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:06.503 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724553 2026-02-25T16:38:06.504 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:06.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:38:06.505 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724553 2026-02-25T16:38:06.505 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:06.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724553 2026-02-25T16:38:06.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724553' 2026-02-25T16:38:06.507 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.8 seq 167503724553 2026-02-25T16:38:06.507 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:38:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724553 -lt 167503724553 2026-02-25T16:38:06.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:06.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-184683593735 2026-02-25T16:38:06.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:06.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:38:06.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-184683593735 2026-02-25T16:38:06.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:06.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593735 2026-02-25T16:38:06.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 184683593735' 2026-02-25T16:38:06.739 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.9 seq 184683593735 2026-02-25T16:38:06.740 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:38:06.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593735 -lt 184683593735 2026-02-25T16:38:06.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:06.964 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462920 2026-02-25T16:38:06.964 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:06.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:38:06.965 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462920 2026-02-25T16:38:06.965 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:06.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462920 2026-02-25T16:38:06.966 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.10 seq 201863462920 2026-02-25T16:38:06.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462920' 2026-02-25T16:38:06.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:38:07.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462920 -lt 201863462920 2026-02-25T16:38:07.193 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:38:07.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:07.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:07.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 16 == 0 2026-02-25T16:38:07.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:38:07.507 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:38:07.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:38:07.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:38:07.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:38:07.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:38:07.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:38:07.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=16 2026-02-25T16:38:07.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:38:07.733 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:07.733 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:08.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 = 16 2026-02-25T16:38:08.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:38:08.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:38:08.045 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2026-02-25T16:38:08.045 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:TEST_chunk_mapping 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_put_get_isa 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_put_get_jerasure 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_put_get_lrc_advanced 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_put_get_lrc_kml 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_put_get_shec' 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: local payload=ABC 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:267: TEST_alignment_constraints: echo ABC 2026-02-25T16:38:08.047 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:273: TEST_alignment_constraints: chunk_size 2026-02-25T16:38:08.048 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T16:38:08.057 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T16:38:08.057 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T16:38:08.058 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T16:38:08.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T16:38:08.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T16:38:08.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T16:38:08.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T16:38:08.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:273: TEST_alignment_constraints: local stripe_unit=4096 2026-02-25T16:38:08.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2026-02-25T16:38:08.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: grep k= 2026-02-25T16:38:08.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: eval local k=2 2026-02-25T16:38:08.305 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:274: TEST_alignment_constraints: local k=2 2026-02-25T16:38:08.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:275: TEST_alignment_constraints: local block_size=8191 2026-02-25T16:38:08.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:276: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2026-02-25T16:38:08.306 INFO:tasks.workunit.client.0.trial129.stderr:2+0 records in 2026-02-25T16:38:08.306 INFO:tasks.workunit.client.0.trial129.stderr:2+0 records out 2026-02-25T16:38:08.306 INFO:tasks.workunit.client.0.trial129.stderr:16382 bytes (16 kB, 16 KiB) copied, 5.8029e-05 s, 282 MB/s 2026-02-25T16:38:08.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2026-02-25T16:38:08.320 INFO:tasks.workunit.client.0.trial129.stderr:INFO: op_size has been rounded to 8192 2026-02-25T16:38:08.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:38:08.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T16:38:08.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2026-02-25T16:38:08.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: local dir=td/test-erasure-code 2026-02-25T16:38:08.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:320: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2026-02-25T16:38:08.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: local dir=td/test-erasure-code 2026-02-25T16:38:08.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local poolname=ecpool 2026-02-25T16:38:08.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: local first=0 2026-02-25T16:38:08.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: local second=1 2026-02-25T16:38:08.326 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T16:38:08.326 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T16:38:08.334 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T16:38:08.334 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T16:38:08.334 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T16:38:08.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T16:38:08.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T16:38:08.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T16:38:08.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T16:38:08.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2026-02-25T16:38:08.344 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T16:38:08.344 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T16:38:08.351 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T16:38:08.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T16:38:08.352 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T16:38:08.362 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T16:38:08.362 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T16:38:08.362 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T16:38:08.362 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T16:38:08.362 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2026-02-25T16:38:08.362 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2026-02-25T16:38:08.363 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2026-02-25T16:38:08.363 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2026-02-25T16:38:08.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2026-02-25T16:38:08.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2026-02-25T16:38:08.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T16:38:08.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2026-02-25T16:38:08.396 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2026-02-25T16:38:08.396 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:08.622 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:38:08.622 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:38:08.622 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:08.623 INFO:tasks.workunit.client.0.trial129.stderr:0' 2026-02-25T16:38:08.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2026-02-25T16:38:08.623 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: osds=('3' '5' '9' '0') 2026-02-25T16:38:08.623 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: local -a osds 2026-02-25T16:38:08.623 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i = 0 )) 2026-02-25T16:38:08.623 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T16:38:08.623 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2026-02-25T16:38:08.673 INFO:tasks.workunit.client.0.trial129.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T16:38:08.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T16:38:08.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T16:38:08.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2026-02-25T16:38:08.726 INFO:tasks.workunit.client.0.trial129.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T16:38:08.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T16:38:08.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T16:38:08.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2026-02-25T16:38:08.778 INFO:tasks.workunit.client.0.trial129.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T16:38:08.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T16:38:08.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T16:38:08.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2026-02-25T16:38:08.829 INFO:tasks.workunit.client.0.trial129.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T16:38:08.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T16:38:08.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 4 )) 2026-02-25T16:38:08.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:38:08.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:305: verify_chunk_mapping: rm td/test-erasure-code/COPY 2026-02-25T16:38:08.832 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2026-02-25T16:38:08.832 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T16:38:08.832 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGecpool 2026-02-25T16:38:08.832 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2026-02-25T16:38:08.833 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:09.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:38:09.061 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:38:09.061 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:09.061 INFO:tasks.workunit.client.0.trial129.stderr:0' 2026-02-25T16:38:09.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 2026-02-25T16:38:09.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: osds=('3' '5' '9' '0') 2026-02-25T16:38:09.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: local -a osds 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: grep --quiet FIRSTecpool 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:38:09.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2026-02-25T16:38:09.063 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:38:09.063 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:38:09.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:38:09.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:38:09.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:38:09.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:38:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-25T16:38:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T16:38:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:38:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T16:38:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:38:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2026-02-25T16:38:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/3 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:38:09.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:09.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:09.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:38:09.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:38:09.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/3 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:start osd.3 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/3/whoami 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:38:09.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:38:09.489 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:09.488+0000 7fef9d848780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:09.491 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:09.489+0000 7fef9d848780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:09.492 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:09.490+0000 7fef9d848780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:09.646 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:09.645+0000 7fef9d848780 -1 Falling back to public interface 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:09.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:38:09.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:38:10.024 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:10.022+0000 7fef9d848780 -1 osd.3 65 log_to_monitors true 2026-02-25T16:38:10.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:38:10.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:10.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:38:10.931 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:10.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:10.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:38:11.156 INFO:tasks.workunit.client.0.trial129.stderr:osd.3 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [18,65) [v2:127.0.0.1:6826/2445285987,v1:127.0.0.1:6827/2445285987] [v2:127.0.0.1:6828/2445285987,v1:127.0.0.1:6829/2445285987] exists,up 64c9513a-0f39-46a2-9911-a4eb3a0d003f 2026-02-25T16:38:11.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:38:11.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:38:11.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:38:11.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:38:11.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:38:11.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:38:11.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:38:11.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:38:11.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:38:11.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:38:11.238 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:38:11.460 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:38:11.461 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:11.461 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:38:11.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:38:11.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:38:11.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836495 2026-02-25T16:38:11.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836495 2026-02-25T16:38:11.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495' 2026-02-25T16:38:11.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.523 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:38:11.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705680 2026-02-25T16:38:11.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705680 2026-02-25T16:38:11.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680' 2026-02-25T16:38:11.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.588 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:38:11.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574862 2026-02-25T16:38:11.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574862 2026-02-25T16:38:11.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862' 2026-02-25T16:38:11.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:38:11.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776131 2026-02-25T16:38:11.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776131 2026-02-25T16:38:11.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131' 2026-02-25T16:38:11.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:38:11.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247821 2026-02-25T16:38:11.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247821 2026-02-25T16:38:11.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131 4-98784247821' 2026-02-25T16:38:11.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:38:11.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117005 2026-02-25T16:38:11.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117005 2026-02-25T16:38:11.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131 4-98784247821 5-115964117005' 2026-02-25T16:38:11.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.844 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:38:11.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986188 2026-02-25T16:38:11.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986188 2026-02-25T16:38:11.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131 4-98784247821 5-115964117005 6-133143986188' 2026-02-25T16:38:11.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.908 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:38:11.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855373 2026-02-25T16:38:11.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855373 2026-02-25T16:38:11.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131 4-98784247821 5-115964117005 6-133143986188 7-150323855373' 2026-02-25T16:38:11.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:11.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:38:12.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724556 2026-02-25T16:38:12.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724556 2026-02-25T16:38:12.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131 4-98784247821 5-115964117005 6-133143986188 7-150323855373 8-167503724556' 2026-02-25T16:38:12.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:12.032 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:38:12.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593739 2026-02-25T16:38:12.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593739 2026-02-25T16:38:12.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131 4-98784247821 5-115964117005 6-133143986188 7-150323855373 8-167503724556 9-184683593739' 2026-02-25T16:38:12.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:12.094 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:38:12.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462923 2026-02-25T16:38:12.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462923 2026-02-25T16:38:12.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836495 1-38654705680 2-55834574862 3-292057776131 4-98784247821 5-115964117005 6-133143986188 7-150323855373 8-167503724556 9-184683593739 10-201863462923' 2026-02-25T16:38:12.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:12.160 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836495 2026-02-25T16:38:12.160 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:12.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:38:12.161 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836495 2026-02-25T16:38:12.161 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:12.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836495 2026-02-25T16:38:12.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2026-02-25T16:38:12.162 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 21474836495 2026-02-25T16:38:12.162 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:12.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836493 -lt 21474836495 2026-02-25T16:38:12.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:38:13.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:38:13.389 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:13.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836495 -lt 21474836495 2026-02-25T16:38:13.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:13.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705680 2026-02-25T16:38:13.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:13.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:38:13.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705680 2026-02-25T16:38:13.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:13.614 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705680 2026-02-25T16:38:13.614 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2026-02-25T16:38:13.615 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 38654705680 2026-02-25T16:38:13.615 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:38:13.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705680 -lt 38654705680 2026-02-25T16:38:13.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:13.839 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574862 2026-02-25T16:38:13.839 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:13.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:38:13.840 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574862 2026-02-25T16:38:13.840 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:13.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574862 2026-02-25T16:38:13.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574862' 2026-02-25T16:38:13.843 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 55834574862 2026-02-25T16:38:13.843 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:38:14.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574862 -lt 55834574862 2026-02-25T16:38:14.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:14.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-292057776131 2026-02-25T16:38:14.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:38:14.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776131 2026-02-25T16:38:14.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:14.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776131 2026-02-25T16:38:14.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 292057776131' 2026-02-25T16:38:14.070 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 292057776131 2026-02-25T16:38:14.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:38:14.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-25T16:38:14.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:14.294 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247821 2026-02-25T16:38:14.294 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:14.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:38:14.295 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247821 2026-02-25T16:38:14.296 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:14.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247821 2026-02-25T16:38:14.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247821' 2026-02-25T16:38:14.297 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.4 seq 98784247821 2026-02-25T16:38:14.297 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:38:14.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247821 -lt 98784247821 2026-02-25T16:38:14.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:14.524 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-115964117005 2026-02-25T16:38:14.525 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:14.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:38:14.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-115964117005 2026-02-25T16:38:14.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:14.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117005 2026-02-25T16:38:14.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 115964117005' 2026-02-25T16:38:14.527 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.5 seq 115964117005 2026-02-25T16:38:14.527 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:38:14.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117005 -lt 115964117005 2026-02-25T16:38:14.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:14.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986188 2026-02-25T16:38:14.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:14.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:38:14.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986188 2026-02-25T16:38:14.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:14.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986188 2026-02-25T16:38:14.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986188' 2026-02-25T16:38:14.755 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.6 seq 133143986188 2026-02-25T16:38:14.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:38:14.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986189 -lt 133143986188 2026-02-25T16:38:14.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:14.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855373 2026-02-25T16:38:14.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:14.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:38:14.981 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855373 2026-02-25T16:38:14.981 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:14.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855373 2026-02-25T16:38:14.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855373' 2026-02-25T16:38:14.983 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.7 seq 150323855373 2026-02-25T16:38:14.983 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:38:15.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855373 -lt 150323855373 2026-02-25T16:38:15.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:15.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724556 2026-02-25T16:38:15.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:15.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:38:15.207 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724556 2026-02-25T16:38:15.207 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:15.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724556 2026-02-25T16:38:15.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724556' 2026-02-25T16:38:15.208 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.8 seq 167503724556 2026-02-25T16:38:15.209 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:38:15.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724557 -lt 167503724556 2026-02-25T16:38:15.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:15.437 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-184683593739 2026-02-25T16:38:15.438 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:38:15.439 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-184683593739 2026-02-25T16:38:15.439 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593739 2026-02-25T16:38:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 184683593739' 2026-02-25T16:38:15.440 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.9 seq 184683593739 2026-02-25T16:38:15.440 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:38:15.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593739 -lt 184683593739 2026-02-25T16:38:15.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:15.665 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462923 2026-02-25T16:38:15.665 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:15.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:38:15.666 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462923 2026-02-25T16:38:15.666 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:15.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462923 2026-02-25T16:38:15.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462923' 2026-02-25T16:38:15.667 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.10 seq 201863462923 2026-02-25T16:38:15.667 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:38:15.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462923 -lt 201863462923 2026-02-25T16:38:15.890 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:38:15.891 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:15.891 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:16.198 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 16 == 0 2026-02-25T16:38:16.198 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:38:16.198 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:38:16.198 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:38:16.198 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:38:16.199 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:38:16.199 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:38:16.199 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:38:16.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=16 2026-02-25T16:38:16.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:38:16.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:16.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:16.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 = 16 2026-02-25T16:38:16.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:38:16.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:38:16.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: grep --quiet SECONDecpool 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-25T16:38:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:38:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2026-02-25T16:38:16.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:38:16.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:38:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:38:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:38:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:38:17.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:38:17.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-25T16:38:17.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T16:38:17.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:38:17.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-25T16:38:17.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:38:17.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2026-02-25T16:38:17.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2026-02-25T16:38:17.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/5 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:38:17.144 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:38:17.145 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:38:17.145 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/5 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:start osd.5 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:38:17.146 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/5/whoami 2026-02-25T16:38:17.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-25T16:38:17.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:38:17.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:38:17.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:38:17.159 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:17.157+0000 7f919602b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:17.160 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:17.158+0000 7f919602b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:17.160 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:17.159+0000 7f919602b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:17.301 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:17.299+0000 7f919602b780 -1 Falling back to public interface 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:17.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:17.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:38:17.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:38:17.634 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:17.632+0000 7f919602b780 -1 osd.5 71 log_to_monitors true 2026-02-25T16:38:18.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:38:18.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:18.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:38:18.599 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:18.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:18.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:38:18.824 INFO:tasks.workunit.client.0.trial129.stderr:osd.5 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [27,71) [v2:127.0.0.1:6842/860402242,v1:127.0.0.1:6843/860402242] [v2:127.0.0.1:6844/860402242,v1:127.0.0.1:6845/860402242] exists,up 46ee556a-e880-40db-ae25-113d5233f602 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:38:18.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:38:18.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:38:18.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:38:18.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:38:18.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:38:18.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:38:18.907 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:38:19.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:38:19.128 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:19.128 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.129 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:38:19.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-25T16:38:19.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-25T16:38:19.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499' 2026-02-25T16:38:19.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:38:19.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705683 2026-02-25T16:38:19.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705683 2026-02-25T16:38:19.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683' 2026-02-25T16:38:19.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.260 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:38:19.322 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574866 2026-02-25T16:38:19.322 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574866 2026-02-25T16:38:19.322 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866' 2026-02-25T16:38:19.323 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.323 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:38:19.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776134 2026-02-25T16:38:19.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776134 2026-02-25T16:38:19.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134' 2026-02-25T16:38:19.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.386 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:38:19.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247825 2026-02-25T16:38:19.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247825 2026-02-25T16:38:19.450 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134 4-98784247825' 2026-02-25T16:38:19.450 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.450 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:38:19.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579907 2026-02-25T16:38:19.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579907 2026-02-25T16:38:19.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134 4-98784247825 5-317827579907' 2026-02-25T16:38:19.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:38:19.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986192 2026-02-25T16:38:19.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986192 2026-02-25T16:38:19.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134 4-98784247825 5-317827579907 6-133143986192' 2026-02-25T16:38:19.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.577 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:38:19.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855376 2026-02-25T16:38:19.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855376 2026-02-25T16:38:19.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134 4-98784247825 5-317827579907 6-133143986192 7-150323855376' 2026-02-25T16:38:19.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.639 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:38:19.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724560 2026-02-25T16:38:19.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724560 2026-02-25T16:38:19.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134 4-98784247825 5-317827579907 6-133143986192 7-150323855376 8-167503724560' 2026-02-25T16:38:19.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.702 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:38:19.763 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593742 2026-02-25T16:38:19.763 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593742 2026-02-25T16:38:19.763 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134 4-98784247825 5-317827579907 6-133143986192 7-150323855376 8-167503724560 9-184683593742' 2026-02-25T16:38:19.763 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:19.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:38:19.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462927 2026-02-25T16:38:19.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462927 2026-02-25T16:38:19.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836499 1-38654705683 2-55834574866 3-292057776134 4-98784247825 5-317827579907 6-133143986192 7-150323855376 8-167503724560 9-184683593742 10-201863462927' 2026-02-25T16:38:19.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:19.827 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836499 2026-02-25T16:38:19.827 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:19.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:38:19.829 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-25T16:38:19.829 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:19.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836499 2026-02-25T16:38:19.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-25T16:38:19.830 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 21474836499 2026-02-25T16:38:19.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:20.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836496 -lt 21474836499 2026-02-25T16:38:20.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:38:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:38:21.058 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:21.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836499 -lt 21474836499 2026-02-25T16:38:21.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:21.283 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705683 2026-02-25T16:38:21.283 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:21.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:38:21.284 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705683 2026-02-25T16:38:21.284 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:21.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705683 2026-02-25T16:38:21.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2026-02-25T16:38:21.285 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 38654705683 2026-02-25T16:38:21.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:38:21.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705683 -lt 38654705683 2026-02-25T16:38:21.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:21.511 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574866 2026-02-25T16:38:21.511 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:21.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:38:21.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574866 2026-02-25T16:38:21.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:21.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574866 2026-02-25T16:38:21.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574866' 2026-02-25T16:38:21.514 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 55834574866 2026-02-25T16:38:21.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:38:21.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574866 -lt 55834574866 2026-02-25T16:38:21.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:21.740 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-292057776134 2026-02-25T16:38:21.740 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:21.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:38:21.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776134 2026-02-25T16:38:21.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:21.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776134 2026-02-25T16:38:21.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 292057776134' 2026-02-25T16:38:21.742 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 292057776134 2026-02-25T16:38:21.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:38:21.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776135 -lt 292057776134 2026-02-25T16:38:21.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:21.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247825 2026-02-25T16:38:21.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:21.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:38:21.974 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247825 2026-02-25T16:38:21.974 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:21.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247825 2026-02-25T16:38:21.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247825' 2026-02-25T16:38:21.975 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.4 seq 98784247825 2026-02-25T16:38:21.975 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:38:22.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247825 -lt 98784247825 2026-02-25T16:38:22.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:22.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-317827579907 2026-02-25T16:38:22.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:22.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:38:22.204 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-317827579907 2026-02-25T16:38:22.204 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:22.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=317827579907 2026-02-25T16:38:22.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 317827579907' 2026-02-25T16:38:22.206 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.5 seq 317827579907 2026-02-25T16:38:22.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:38:22.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 317827579907 -lt 317827579907 2026-02-25T16:38:22.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:22.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-133143986192 2026-02-25T16:38:22.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:22.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:38:22.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-133143986192 2026-02-25T16:38:22.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:22.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=133143986192 2026-02-25T16:38:22.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 133143986192' 2026-02-25T16:38:22.437 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.6 seq 133143986192 2026-02-25T16:38:22.437 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:38:22.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 133143986192 -lt 133143986192 2026-02-25T16:38:22.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:22.660 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855376 2026-02-25T16:38:22.660 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:22.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:38:22.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855376 2026-02-25T16:38:22.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:22.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855376 2026-02-25T16:38:22.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855376' 2026-02-25T16:38:22.663 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.7 seq 150323855376 2026-02-25T16:38:22.663 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:38:22.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855377 -lt 150323855376 2026-02-25T16:38:22.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:22.889 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724560 2026-02-25T16:38:22.889 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:22.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:38:22.890 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724560 2026-02-25T16:38:22.891 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:22.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724560 2026-02-25T16:38:22.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724560' 2026-02-25T16:38:22.892 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.8 seq 167503724560 2026-02-25T16:38:22.892 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:38:23.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724560 -lt 167503724560 2026-02-25T16:38:23.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:23.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-184683593742 2026-02-25T16:38:23.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:23.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:38:23.118 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-184683593742 2026-02-25T16:38:23.118 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:23.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593742 2026-02-25T16:38:23.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 184683593742' 2026-02-25T16:38:23.119 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.9 seq 184683593742 2026-02-25T16:38:23.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:38:23.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593743 -lt 184683593742 2026-02-25T16:38:23.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:23.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462927 2026-02-25T16:38:23.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:23.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:38:23.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462927 2026-02-25T16:38:23.345 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:23.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462927 2026-02-25T16:38:23.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462927' 2026-02-25T16:38:23.346 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.10 seq 201863462927 2026-02-25T16:38:23.346 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:38:23.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462927 -lt 201863462927 2026-02-25T16:38:23.569 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:38:23.569 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:23.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:23.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 16 == 0 2026-02-25T16:38:23.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:38:23.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:38:23.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:38:23.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:38:23.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:38:23.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:38:23.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:38:24.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=16 2026-02-25T16:38:24.108 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:38:24.108 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:24.108 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:24.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 = 16 2026-02-25T16:38:24.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:38:24.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:38:24.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:322: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2026-02-25T16:38:24.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2026-02-25T16:38:24.961 INFO:tasks.workunit.client.0.trial129.stdout:crush-device-class= 2026-02-25T16:38:24.961 INFO:tasks.workunit.client.0.trial129.stdout:crush-failure-domain=host 2026-02-25T16:38:24.961 INFO:tasks.workunit.client.0.trial129.stdout:crush-num-failure-domains=0 2026-02-25T16:38:24.961 INFO:tasks.workunit.client.0.trial129.stdout:crush-osds-per-failure-domain=0 2026-02-25T16:38:24.961 INFO:tasks.workunit.client.0.trial129.stdout:crush-root=default 2026-02-25T16:38:24.961 INFO:tasks.workunit.client.0.trial129.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2026-02-25T16:38:24.961 INFO:tasks.workunit.client.0.trial129.stdout:k=-1 2026-02-25T16:38:24.962 INFO:tasks.workunit.client.0.trial129.stdout:l=-1 2026-02-25T16:38:24.962 INFO:tasks.workunit.client.0.trial129.stdout:layers=[ [ "cDD", "" ] ] 2026-02-25T16:38:24.962 INFO:tasks.workunit.client.0.trial129.stdout:m=-1 2026-02-25T16:38:24.962 INFO:tasks.workunit.client.0.trial129.stdout:mapping=_DD 2026-02-25T16:38:24.962 INFO:tasks.workunit.client.0.trial129.stdout:plugin=lrc 2026-02-25T16:38:24.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:328: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2026-02-25T16:38:24.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2026-02-25T16:38:25.313 INFO:tasks.workunit.client.0.trial129.stderr:pool 'remap-pool' already exists 2026-02-25T16:38:25.322 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:38:26.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:336: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2026-02-25T16:38:26.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: local dir=td/test-erasure-code 2026-02-25T16:38:26.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local poolname=remap-pool 2026-02-25T16:38:26.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: local first=1 2026-02-25T16:38:26.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: local second=2 2026-02-25T16:38:26.325 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T16:38:26.325 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T16:38:26.333 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T16:38:26.334 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T16:38:26.334 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T16:38:26.342 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T16:38:26.342 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T16:38:26.342 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T16:38:26.342 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T16:38:26.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2026-02-25T16:38:26.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: chunk_size 2026-02-25T16:38:26.343 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-25T16:38:26.351 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: chunk_size: local chunk_size=0 2026-02-25T16:38:26.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: chunk_size: '[' 0 -eq 0 ']' 2026-02-25T16:38:26.352 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: ceph-conf --show-config-value osd_pool_default_flag_ec_optimizations 2026-02-25T16:38:26.360 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:255: chunk_size: local ec_opt=false 2026-02-25T16:38:26.360 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: chunk_size: '[' false = true ']' 2026-02-25T16:38:26.360 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: chunk_size: chunk_size=4096 2026-02-25T16:38:26.360 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: chunk_size: echo 4096 2026-02-25T16:38:26.360 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2026-02-25T16:38:26.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2026-02-25T16:38:26.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2026-02-25T16:38:26.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2026-02-25T16:38:26.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2026-02-25T16:38:26.395 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2026-02-25T16:38:26.395 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2026-02-25T16:38:26.395 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2026-02-25T16:38:26.395 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2026-02-25T16:38:26.395 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:26.621 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T16:38:26.621 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:38:26.621 INFO:tasks.workunit.client.0.trial129.stderr:9' 2026-02-25T16:38:26.621 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2026-02-25T16:38:26.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: osds=('4' '6' '9') 2026-02-25T16:38:26.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: local -a osds 2026-02-25T16:38:26.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i = 0 )) 2026-02-25T16:38:26.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T16:38:26.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2026-02-25T16:38:26.671 INFO:tasks.workunit.client.0.trial129.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T16:38:26.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T16:38:26.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T16:38:26.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2026-02-25T16:38:26.723 INFO:tasks.workunit.client.0.trial129.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T16:38:26.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T16:38:26.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T16:38:26.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:302: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2026-02-25T16:38:26.773 INFO:tasks.workunit.client.0.trial129.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-25T16:38:26.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i++ )) 2026-02-25T16:38:26.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:301: verify_chunk_mapping: (( i < 3 )) 2026-02-25T16:38:26.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:38:26.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:305: verify_chunk_mapping: rm td/test-erasure-code/COPY 2026-02-25T16:38:26.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2026-02-25T16:38:26.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=remap-pool 2026-02-25T16:38:26.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHINGremap-pool 2026-02-25T16:38:26.777 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2026-02-25T16:38:26.777 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:27.009 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T16:38:27.009 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:38:27.009 INFO:tasks.workunit.client.0.trial129.stderr:9' 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 6 9 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: osds=('4' '6' '9') 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: verify_chunk_mapping: local -a osds 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:308: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=6 2026-02-25T16:38:27.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=6 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:38:27.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:38:27.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:38:27.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:27.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T16:38:27.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:38:27.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=6 2026-02-25T16:38:27.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:38:27.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2026-02-25T16:38:27.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:27.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2026-02-25T16:38:27.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T16:38:27.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:38:27.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=6 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/6 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:38:27.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:38:27.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/6 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.6 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:start osd.6 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 6 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/6/whoami 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 6 = 6 ']' 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:38:27.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:38:27.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:38:27.444 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:27.443+0000 7fb955e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:27.446 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:27.444+0000 7fb955e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:27.447 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:27.445+0000 7fb955e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:27.585 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:27.583+0000 7fb955e39780 -1 Falling back to public interface 2026-02-25T16:38:27.671 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 6 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:27.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:38:27.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:38:27.964 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:27.962+0000 7fb955e39780 -1 osd.6 81 log_to_monitors true 2026-02-25T16:38:28.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:38:28.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:28.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:38:28.896 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:28.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:28.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:osd.6 up in weight 1 up_from 85 up_thru 85 down_at 82 last_clean_interval [31,81) [v2:127.0.0.1:6850/3130843022,v1:127.0.0.1:6851/3130843022] [v2:127.0.0.1:6852/3130843022,v1:127.0.0.1:6853/3130843022] exists,up 4a2a6f51-6e33-4f14-8ee5-ef90410a7cbc 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:38:29.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:38:29.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:38:29.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:38:29.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:38:29.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:38:29.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:38:29.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:38:29.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:38:29.437 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:29.437 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:38:29.437 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:38:29.437 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:38:29.437 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:38:29.437 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:38:29.437 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:38:29.438 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:38:29.438 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:29.438 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:38:29.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:38:29.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.438 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:38:29.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-25T16:38:29.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-25T16:38:29.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503' 2026-02-25T16:38:29.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.504 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:38:29.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705687 2026-02-25T16:38:29.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705687 2026-02-25T16:38:29.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687' 2026-02-25T16:38:29.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.568 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:38:29.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574870 2026-02-25T16:38:29.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574870 2026-02-25T16:38:29.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870' 2026-02-25T16:38:29.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.633 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:38:29.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776138 2026-02-25T16:38:29.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776138 2026-02-25T16:38:29.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138' 2026-02-25T16:38:29.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:38:29.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247829 2026-02-25T16:38:29.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247829 2026-02-25T16:38:29.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138 4-98784247829' 2026-02-25T16:38:29.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:38:29.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579911 2026-02-25T16:38:29.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579911 2026-02-25T16:38:29.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138 4-98784247829 5-317827579911' 2026-02-25T16:38:29.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:38:29.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220163 2026-02-25T16:38:29.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220163 2026-02-25T16:38:29.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138 4-98784247829 5-317827579911 6-365072220163' 2026-02-25T16:38:29.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.885 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:38:29.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855380 2026-02-25T16:38:29.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855380 2026-02-25T16:38:29.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138 4-98784247829 5-317827579911 6-365072220163 7-150323855380' 2026-02-25T16:38:29.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:29.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:38:30.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724564 2026-02-25T16:38:30.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724564 2026-02-25T16:38:30.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138 4-98784247829 5-317827579911 6-365072220163 7-150323855380 8-167503724564' 2026-02-25T16:38:30.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:30.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:38:30.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593746 2026-02-25T16:38:30.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593746 2026-02-25T16:38:30.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138 4-98784247829 5-317827579911 6-365072220163 7-150323855380 8-167503724564 9-184683593746' 2026-02-25T16:38:30.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:30.079 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:38:30.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462931 2026-02-25T16:38:30.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462931 2026-02-25T16:38:30.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836503 1-38654705687 2-55834574870 3-292057776138 4-98784247829 5-317827579911 6-365072220163 7-150323855380 8-167503724564 9-184683593746 10-201863462931' 2026-02-25T16:38:30.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:30.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836503 2026-02-25T16:38:30.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:30.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:38:30.148 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-25T16:38:30.148 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:30.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836503 2026-02-25T16:38:30.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-25T16:38:30.149 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 21474836503 2026-02-25T16:38:30.149 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:30.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836500 -lt 21474836503 2026-02-25T16:38:30.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:38:31.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:38:31.376 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:31.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836503 -lt 21474836503 2026-02-25T16:38:31.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:31.599 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705687 2026-02-25T16:38:31.599 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:31.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:38:31.600 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705687 2026-02-25T16:38:31.600 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:31.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705687 2026-02-25T16:38:31.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705687' 2026-02-25T16:38:31.602 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 38654705687 2026-02-25T16:38:31.603 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:38:31.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705687 -lt 38654705687 2026-02-25T16:38:31.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:31.826 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574870 2026-02-25T16:38:31.826 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:31.826 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:38:31.827 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574870 2026-02-25T16:38:31.827 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:31.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574870 2026-02-25T16:38:31.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574870' 2026-02-25T16:38:31.828 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 55834574870 2026-02-25T16:38:31.828 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:38:32.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574870 -lt 55834574870 2026-02-25T16:38:32.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:32.052 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-292057776138 2026-02-25T16:38:32.052 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:32.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:38:32.053 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776138 2026-02-25T16:38:32.054 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:32.055 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776138 2026-02-25T16:38:32.055 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 292057776138' 2026-02-25T16:38:32.055 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 292057776138 2026-02-25T16:38:32.055 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:38:32.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776139 -lt 292057776138 2026-02-25T16:38:32.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:32.280 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247829 2026-02-25T16:38:32.280 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:32.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:38:32.281 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247829 2026-02-25T16:38:32.281 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:32.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247829 2026-02-25T16:38:32.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247829' 2026-02-25T16:38:32.282 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.4 seq 98784247829 2026-02-25T16:38:32.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:38:32.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247829 -lt 98784247829 2026-02-25T16:38:32.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:32.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-317827579911 2026-02-25T16:38:32.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:32.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:38:32.509 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-317827579911 2026-02-25T16:38:32.510 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=317827579911 2026-02-25T16:38:32.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 317827579911' 2026-02-25T16:38:32.511 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.5 seq 317827579911 2026-02-25T16:38:32.511 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:38:32.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 317827579911 -lt 317827579911 2026-02-25T16:38:32.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:32.739 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-365072220163 2026-02-25T16:38:32.739 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:32.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:38:32.740 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-365072220163 2026-02-25T16:38:32.740 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:32.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=365072220163 2026-02-25T16:38:32.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 365072220163' 2026-02-25T16:38:32.741 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.6 seq 365072220163 2026-02-25T16:38:32.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:38:32.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 365072220163 -lt 365072220163 2026-02-25T16:38:32.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:32.966 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855380 2026-02-25T16:38:32.966 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:32.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:38:32.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855380 2026-02-25T16:38:32.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:32.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855380 2026-02-25T16:38:32.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855380' 2026-02-25T16:38:32.969 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.7 seq 150323855380 2026-02-25T16:38:32.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:38:33.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855381 -lt 150323855380 2026-02-25T16:38:33.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:33.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724564 2026-02-25T16:38:33.193 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:33.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:38:33.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724564 2026-02-25T16:38:33.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:33.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724564 2026-02-25T16:38:33.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724564' 2026-02-25T16:38:33.195 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.8 seq 167503724564 2026-02-25T16:38:33.196 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:38:33.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724564 -lt 167503724564 2026-02-25T16:38:33.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:33.421 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-184683593746 2026-02-25T16:38:33.421 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:33.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:38:33.422 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-184683593746 2026-02-25T16:38:33.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:33.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593746 2026-02-25T16:38:33.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 184683593746' 2026-02-25T16:38:33.424 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.9 seq 184683593746 2026-02-25T16:38:33.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:38:33.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593747 -lt 184683593746 2026-02-25T16:38:33.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:33.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462931 2026-02-25T16:38:33.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:33.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:38:33.653 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462931 2026-02-25T16:38:33.653 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:33.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462931 2026-02-25T16:38:33.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462931' 2026-02-25T16:38:33.654 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.10 seq 201863462931 2026-02-25T16:38:33.655 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:38:33.878 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462931 -lt 201863462931 2026-02-25T16:38:33.879 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:38:33.879 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:33.879 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:34.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T16:38:34.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:38:34.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:38:34.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:38:34.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:38:34.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:38:34.195 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:38:34.195 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:38:34.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T16:38:34.421 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:38:34.421 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:34.421 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:34.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-code 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:38:34.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: verify_chunk_mapping: grep --quiet SECONDremap-pool 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:38:34.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:38:34.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:38:34.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:38:34.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:34.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-25T16:38:34.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:38:34.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-25T16:38:34.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:38:34.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2026-02-25T16:38:34.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2026-02-25T16:38:34.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2026-02-25T16:38:34.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-code 2026-02-25T16:38:34.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:38:34.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-code/9 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-25T16:38:34.952 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:38:34.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-code/9 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:start osd.9 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=b4ea5cef-1771-473e-9d6d-7cf8a50c246b --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.68264/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:38:34.954 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-code/9/whoami 2026-02-25T16:38:34.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-25T16:38:34.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:38:34.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:38:34.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:38:34.964 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:34.962+0000 7f834f63d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:34.965 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:34.963+0000 7f834f63d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:34.966 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:34.964+0000 7f834f63d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:38:35.168 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:35.166+0000 7f834f63d780 -1 Falling back to public interface 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:38:35.177 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:35.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:35.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T16:38:35.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:38:35.539 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:38:35.537+0000 7f834f63d780 -1 osd.9 87 log_to_monitors true 2026-02-25T16:38:36.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:38:36.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:38:36.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:38:36.404 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:36.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:38:36.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-25T16:38:36.630 INFO:tasks.workunit.client.0.trial129.stderr:osd.9 up in weight 1 up_from 91 up_thru 91 down_at 88 last_clean_interval [43,87) [v2:127.0.0.1:6874/1411551140,v1:127.0.0.1:6875/1411551140] [v2:127.0.0.1:6876/1411551140,v1:127.0.0.1:6877/1411551140] exists,up e2ab4810-4d41-4861-8151-2b72f14a40df 2026-02-25T16:38:36.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:38:36.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:38:36.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:38:36.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:38:36.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:38:36.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:38:36.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:38:36.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:38:36.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:38:36.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:38:36.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:38:36.710 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:38:36.933 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:38:36.934 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:36.934 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:38:36.934 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:38:36.934 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:36.934 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:38:36.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836506 2026-02-25T16:38:36.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836506 2026-02-25T16:38:36.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506' 2026-02-25T16:38:36.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:36.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:38:37.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705691 2026-02-25T16:38:37.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705691 2026-02-25T16:38:37.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691' 2026-02-25T16:38:37.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.063 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:38:37.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574873 2026-02-25T16:38:37.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574873 2026-02-25T16:38:37.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873' 2026-02-25T16:38:37.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:38:37.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776142 2026-02-25T16:38:37.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776142 2026-02-25T16:38:37.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142' 2026-02-25T16:38:37.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.193 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:38:37.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247832 2026-02-25T16:38:37.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247832 2026-02-25T16:38:37.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142 4-98784247832' 2026-02-25T16:38:37.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.257 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:38:37.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579914 2026-02-25T16:38:37.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579914 2026-02-25T16:38:37.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142 4-98784247832 5-317827579914' 2026-02-25T16:38:37.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.320 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:38:37.382 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220166 2026-02-25T16:38:37.382 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220166 2026-02-25T16:38:37.382 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142 4-98784247832 5-317827579914 6-365072220166' 2026-02-25T16:38:37.382 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.382 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:38:37.446 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855384 2026-02-25T16:38:37.446 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855384 2026-02-25T16:38:37.446 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142 4-98784247832 5-317827579914 6-365072220166 7-150323855384' 2026-02-25T16:38:37.446 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.446 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:38:37.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724567 2026-02-25T16:38:37.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724567 2026-02-25T16:38:37.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142 4-98784247832 5-317827579914 6-365072220166 7-150323855384 8-167503724567' 2026-02-25T16:38:37.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.511 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:38:37.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023939 2026-02-25T16:38:37.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023939 2026-02-25T16:38:37.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142 4-98784247832 5-317827579914 6-365072220166 7-150323855384 8-167503724567 9-390842023939' 2026-02-25T16:38:37.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:38:37.574 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:38:37.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462934 2026-02-25T16:38:37.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462934 2026-02-25T16:38:37.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836506 1-38654705691 2-55834574873 3-292057776142 4-98784247832 5-317827579914 6-365072220166 7-150323855384 8-167503724567 9-390842023939 10-201863462934' 2026-02-25T16:38:37.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:37.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836506 2026-02-25T16:38:37.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:37.637 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:38:37.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836506 2026-02-25T16:38:37.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:37.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836506 2026-02-25T16:38:37.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2026-02-25T16:38:37.639 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 21474836506 2026-02-25T16:38:37.640 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:37.863 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836504 -lt 21474836506 2026-02-25T16:38:37.863 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:38:38.864 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:38:38.864 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:38:39.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836506 -lt 21474836506 2026-02-25T16:38:39.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:39.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705691 2026-02-25T16:38:39.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:39.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:38:39.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705691 2026-02-25T16:38:39.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:39.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705691 2026-02-25T16:38:39.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705691' 2026-02-25T16:38:39.091 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 38654705691 2026-02-25T16:38:39.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:38:39.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705691 -lt 38654705691 2026-02-25T16:38:39.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:39.315 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574873 2026-02-25T16:38:39.316 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:39.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:38:39.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574873 2026-02-25T16:38:39.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:39.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574873 2026-02-25T16:38:39.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574873' 2026-02-25T16:38:39.318 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 55834574873 2026-02-25T16:38:39.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:38:39.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574873 -lt 55834574873 2026-02-25T16:38:39.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:39.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-292057776142 2026-02-25T16:38:39.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:39.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:38:39.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776142 2026-02-25T16:38:39.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:39.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776142 2026-02-25T16:38:39.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 292057776142' 2026-02-25T16:38:39.549 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 292057776142 2026-02-25T16:38:39.549 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:38:39.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776142 -lt 292057776142 2026-02-25T16:38:39.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:39.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247832 2026-02-25T16:38:39.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:39.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:38:39.778 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247832 2026-02-25T16:38:39.778 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:39.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247832 2026-02-25T16:38:39.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247832' 2026-02-25T16:38:39.779 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.4 seq 98784247832 2026-02-25T16:38:39.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:38:40.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247832 -lt 98784247832 2026-02-25T16:38:40.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:40.004 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-317827579914 2026-02-25T16:38:40.004 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:40.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:38:40.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-317827579914 2026-02-25T16:38:40.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:40.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=317827579914 2026-02-25T16:38:40.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 317827579914' 2026-02-25T16:38:40.006 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.5 seq 317827579914 2026-02-25T16:38:40.007 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:38:40.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 317827579915 -lt 317827579914 2026-02-25T16:38:40.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:40.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-365072220166 2026-02-25T16:38:40.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:40.235 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:38:40.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-365072220166 2026-02-25T16:38:40.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:40.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=365072220166 2026-02-25T16:38:40.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 365072220166' 2026-02-25T16:38:40.237 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.6 seq 365072220166 2026-02-25T16:38:40.237 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:38:40.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 365072220167 -lt 365072220166 2026-02-25T16:38:40.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:40.464 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855384 2026-02-25T16:38:40.464 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:40.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:38:40.465 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855384 2026-02-25T16:38:40.465 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:40.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855384 2026-02-25T16:38:40.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855384' 2026-02-25T16:38:40.466 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.7 seq 150323855384 2026-02-25T16:38:40.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:38:40.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855384 -lt 150323855384 2026-02-25T16:38:40.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:40.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724567 2026-02-25T16:38:40.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:40.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:38:40.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724567 2026-02-25T16:38:40.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:40.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724567 2026-02-25T16:38:40.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724567' 2026-02-25T16:38:40.697 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.8 seq 167503724567 2026-02-25T16:38:40.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:38:40.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724568 -lt 167503724567 2026-02-25T16:38:40.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:40.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-390842023939 2026-02-25T16:38:40.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:40.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:38:40.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-390842023939 2026-02-25T16:38:40.927 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:40.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=390842023939 2026-02-25T16:38:40.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 390842023939' 2026-02-25T16:38:40.928 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.9 seq 390842023939 2026-02-25T16:38:40.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:38:41.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 390842023939 -lt 390842023939 2026-02-25T16:38:41.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:38:41.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462934 2026-02-25T16:38:41.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:38:41.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:38:41.155 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462934 2026-02-25T16:38:41.156 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:38:41.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462934 2026-02-25T16:38:41.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462934' 2026-02-25T16:38:41.156 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.10 seq 201863462934 2026-02-25T16:38:41.157 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:38:41.383 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462935 -lt 201863462934 2026-02-25T16:38:41.383 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:38:41.383 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:41.383 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:41.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T16:38:41.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:38:41.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:38:41.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:38:41.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:38:41.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:38:41.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:38:41.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:38:41.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T16:38:41.915 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:38:41.915 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:38:41.915 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:38:42.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T16:38:42.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:38:42.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:38:42.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:338: TEST_chunk_mapping: delete_pool remap-pool 2026-02-25T16:38:42.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=remap-pool 2026-02-25T16:38:42.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2026-02-25T16:38:42.507 INFO:tasks.workunit.client.0.trial129.stderr:pool 'remap-pool' does not exist 2026-02-25T16:38:42.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:339: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2026-02-25T16:38:42.817 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile remap-profile does not exist 2026-02-25T16:38:42.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T16:38:42.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2026-02-25T16:38:42.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2026-02-25T16:38:42.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=isa 2026-02-25T16:38:42.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-25T16:38:42.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-25T16:38:42.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-25T16:38:42.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-25T16:38:42.828 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-25T16:38:42.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='isa.*No such file' 2026-02-25T16:38:42.829 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2026-02-25T16:38:43.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s= 2026-02-25T16:38:43.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=0 2026-02-25T16:38:43.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-25T16:38:43.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-25T16:38:43.389 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-25T16:38:43.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 0 2026-02-25T16:38:43.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2026-02-25T16:38:43.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2026-02-25T16:38:43.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2026-02-25T16:38:43.683 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2026-02-25T16:38:43.683 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2026-02-25T16:38:44.051 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-isa' already exists 2026-02-25T16:38:44.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:38:45.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2026-02-25T16:38:45.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T16:38:45.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2026-02-25T16:38:45.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T16:38:45.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T16:38:45.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:38:45.098 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:38:45.099 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T16:38:45.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2026-02-25T16:38:45.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2026-02-25T16:38:45.102 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:38:45.102 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2026-02-25T16:38:45.102 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:45.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T16:38:45.330 INFO:tasks.workunit.client.0.trial129.stderr:10 2026-02-25T16:38:45.330 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:45.330 INFO:tasks.workunit.client.0.trial129.stderr:7' 2026-02-25T16:38:45.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 7 2026-02-25T16:38:45.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7') 2026-02-25T16:38:45.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T16:38:45.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2026-02-25T16:38:45.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2026-02-25T16:38:45.578 INFO:tasks.workunit.client.0.trial129.stderr:osd.7 is already out. 2026-02-25T16:38:45.586 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T16:38:50.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2026-02-25T16:38:50.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-isa 2026-02-25T16:38:50.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:38:50.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2026-02-25T16:38:50.587 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-isa SOMETHING 2026-02-25T16:38:50.588 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:50.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=4 2026-02-25T16:38:50.814 INFO:tasks.workunit.client.0.trial129.stderr:10 2026-02-25T16:38:50.815 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:50.815 INFO:tasks.workunit.client.0.trial129.stderr:1' 2026-02-25T16:38:50.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 4 10 0 1 2026-02-25T16:38:50.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:38:50.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:38:50.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2026-02-25T16:38:51.063 INFO:tasks.workunit.client.0.trial129.stderr:osd.7 is already in. 2026-02-25T16:38:51.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:38:51.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2026-02-25T16:38:51.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-isa 2026-02-25T16:38:51.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2026-02-25T16:38:51.363 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-isa' does not exist 2026-02-25T16:38:51.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T16:38:51.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2026-02-25T16:38:51.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2026-02-25T16:38:51.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2026-02-25T16:38:51.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T16:38:51.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T16:38:51.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:38:51.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:38:51.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T16:38:51.410 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2026-02-25T16:38:51.410 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T16:38:51.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:38:51.411 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-25T16:38:51.411 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:51.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr:9' 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 9 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2026-02-25T16:38:51.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2026-02-25T16:38:51.888 INFO:tasks.workunit.client.0.trial129.stderr:osd.9 is already out. 2026-02-25T16:38:51.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T16:38:56.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2026-02-25T16:38:56.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-25T16:38:56.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:38:56.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2026-02-25T16:38:56.897 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-25T16:38:56.897 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:57.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=6 2026-02-25T16:38:57.126 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:38:57.126 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:38:57.126 INFO:tasks.workunit.client.0.trial129.stderr:5' 2026-02-25T16:38:57.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 6 0 7 5 2026-02-25T16:38:57.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:38:57.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:38:57.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2026-02-25T16:38:57.374 INFO:tasks.workunit.client.0.trial129.stderr:osd.9 is already in. 2026-02-25T16:38:57.382 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:38:57.383 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2026-02-25T16:38:57.383 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2026-02-25T16:38:57.383 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2026-02-25T16:38:57.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2026-02-25T16:38:57.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2026-02-25T16:38:58.062 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:38:58.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:38:59.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2026-02-25T16:38:59.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T16:38:59.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2026-02-25T16:38:59.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T16:38:59.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T16:38:59.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T16:38:59.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:38:59.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:38:59.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T16:38:59.111 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2026-02-25T16:38:59.111 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:38:59.111 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:38:59.112 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T16:38:59.112 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:38:59.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T16:38:59.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2026-02-25T16:38:59.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2026-02-25T16:38:59.602 INFO:tasks.workunit.client.0.trial129.stderr:osd.10 is already out. 2026-02-25T16:38:59.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T16:39:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2026-02-25T16:39:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:39:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:04.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2026-02-25T16:39:04.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T16:39:04.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:04.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T16:39:04.850 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:39:04.850 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:04.850 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:04.850 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:04.850 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:39:04.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 6 2026-02-25T16:39:04.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:04.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:04.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2026-02-25T16:39:05.148 INFO:tasks.workunit.client.0.trial129.stderr:osd.10 is already in. 2026-02-25T16:39:05.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:39:05.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2026-02-25T16:39:05.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2026-02-25T16:39:05.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2026-02-25T16:39:05.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2026-02-25T16:39:05.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T16:39:05.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:05.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:05.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2026-02-25T16:39:05.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2026-02-25T16:39:05.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:39:05.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:39:05.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:39:05.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:39:05.485 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:39:05.485 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:39:05.485 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:39:05.485 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:39:05.485 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:39:05.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:39:05.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:39:05.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:39:05.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:39:05.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:39:05.566 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:39:05.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:05.789 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:39:05.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-25T16:39:05.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-25T16:39:05.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514' 2026-02-25T16:39:05.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:05.852 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:39:05.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705698 2026-02-25T16:39:05.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705698 2026-02-25T16:39:05.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698' 2026-02-25T16:39:05.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:05.922 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:39:05.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574881 2026-02-25T16:39:05.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574881 2026-02-25T16:39:05.985 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881' 2026-02-25T16:39:05.985 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:05.985 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:39:06.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776150 2026-02-25T16:39:06.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776150 2026-02-25T16:39:06.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150' 2026-02-25T16:39:06.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:06.048 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:39:06.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247840 2026-02-25T16:39:06.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247840 2026-02-25T16:39:06.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150 4-98784247840' 2026-02-25T16:39:06.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:06.110 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:39:06.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579922 2026-02-25T16:39:06.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579922 2026-02-25T16:39:06.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150 4-98784247840 5-317827579922' 2026-02-25T16:39:06.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:06.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:39:06.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220174 2026-02-25T16:39:06.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220174 2026-02-25T16:39:06.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150 4-98784247840 5-317827579922 6-365072220174' 2026-02-25T16:39:06.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:06.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:39:06.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855391 2026-02-25T16:39:06.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855391 2026-02-25T16:39:06.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150 4-98784247840 5-317827579922 6-365072220174 7-150323855391' 2026-02-25T16:39:06.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:06.303 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:39:06.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724575 2026-02-25T16:39:06.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724575 2026-02-25T16:39:06.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150 4-98784247840 5-317827579922 6-365072220174 7-150323855391 8-167503724575' 2026-02-25T16:39:06.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:06.366 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:39:06.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023947 2026-02-25T16:39:06.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023947 2026-02-25T16:39:06.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150 4-98784247840 5-317827579922 6-365072220174 7-150323855391 8-167503724575 9-390842023947' 2026-02-25T16:39:06.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:06.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:39:06.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462942 2026-02-25T16:39:06.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462942 2026-02-25T16:39:06.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836514 1-38654705698 2-55834574881 3-292057776150 4-98784247840 5-317827579922 6-365072220174 7-150323855391 8-167503724575 9-390842023947 10-201863462942' 2026-02-25T16:39:06.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:06.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836514 2026-02-25T16:39:06.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:06.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:39:06.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-25T16:39:06.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:06.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836514 2026-02-25T16:39:06.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-25T16:39:06.498 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 21474836514 2026-02-25T16:39:06.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:39:06.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836514 -lt 21474836514 2026-02-25T16:39:06.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:06.720 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705698 2026-02-25T16:39:06.720 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:06.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:39:06.722 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705698 2026-02-25T16:39:06.722 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:06.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705698 2026-02-25T16:39:06.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705698' 2026-02-25T16:39:06.723 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 38654705698 2026-02-25T16:39:06.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:39:06.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705699 -lt 38654705698 2026-02-25T16:39:06.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:06.946 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574881 2026-02-25T16:39:06.946 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:06.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:39:06.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574881 2026-02-25T16:39:06.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:06.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574881 2026-02-25T16:39:06.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574881' 2026-02-25T16:39:06.949 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 55834574881 2026-02-25T16:39:06.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:39:07.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574881 -lt 55834574881 2026-02-25T16:39:07.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:07.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-292057776150 2026-02-25T16:39:07.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:07.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:39:07.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776150 2026-02-25T16:39:07.180 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:07.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776150 2026-02-25T16:39:07.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 292057776150' 2026-02-25T16:39:07.181 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 292057776150 2026-02-25T16:39:07.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:39:07.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776150 -lt 292057776150 2026-02-25T16:39:07.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:07.403 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247840 2026-02-25T16:39:07.403 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:07.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:39:07.404 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247840 2026-02-25T16:39:07.404 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:07.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247840 2026-02-25T16:39:07.406 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 98784247840 2026-02-25T16:39:07.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247840' 2026-02-25T16:39:07.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:39:07.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247840 -lt 98784247840 2026-02-25T16:39:07.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:07.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-317827579922 2026-02-25T16:39:07.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:07.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:39:07.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-317827579922 2026-02-25T16:39:07.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:07.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=317827579922 2026-02-25T16:39:07.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 317827579922' 2026-02-25T16:39:07.634 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 317827579922 2026-02-25T16:39:07.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:39:07.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 317827579922 -lt 317827579922 2026-02-25T16:39:07.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:07.859 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-365072220174 2026-02-25T16:39:07.859 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:07.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:39:07.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-365072220174 2026-02-25T16:39:07.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:07.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=365072220174 2026-02-25T16:39:07.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 365072220174' 2026-02-25T16:39:07.862 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 365072220174 2026-02-25T16:39:07.862 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:39:08.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 365072220174 -lt 365072220174 2026-02-25T16:39:08.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:08.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855391 2026-02-25T16:39:08.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:08.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:39:08.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855391 2026-02-25T16:39:08.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:08.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855391 2026-02-25T16:39:08.091 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.7 seq 150323855391 2026-02-25T16:39:08.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855391' 2026-02-25T16:39:08.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:39:08.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855391 -lt 150323855391 2026-02-25T16:39:08.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:08.316 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724575 2026-02-25T16:39:08.316 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:08.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:39:08.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724575 2026-02-25T16:39:08.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:08.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724575 2026-02-25T16:39:08.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724575' 2026-02-25T16:39:08.319 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.8 seq 167503724575 2026-02-25T16:39:08.320 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:39:08.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724575 -lt 167503724575 2026-02-25T16:39:08.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:08.544 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-390842023947 2026-02-25T16:39:08.544 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:08.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:39:08.545 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-390842023947 2026-02-25T16:39:08.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:08.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=390842023947 2026-02-25T16:39:08.547 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.9 seq 390842023947 2026-02-25T16:39:08.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 390842023947' 2026-02-25T16:39:08.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:39:08.772 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 390842023947 -lt 390842023947 2026-02-25T16:39:08.772 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:08.773 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462942 2026-02-25T16:39:08.773 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:08.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:39:08.774 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462942 2026-02-25T16:39:08.774 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:08.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462942 2026-02-25T16:39:08.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462942' 2026-02-25T16:39:08.775 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.10 seq 201863462942 2026-02-25T16:39:08.776 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:39:08.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462942 -lt 201863462942 2026-02-25T16:39:08.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:39:08.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:39:08.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:39:09.312 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T16:39:09.312 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:39:09.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:39:09.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:39:09.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:39:09.312 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:39:09.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:39:09.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:39:09.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T16:39:09.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:39:09.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:39:09.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:39:09.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T16:39:09.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:39:09.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:39:09.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T16:39:09.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:39:09.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:09.848 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T16:39:09.848 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:10.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T16:39:10.073 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:39:10.073 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2026-02-25T16:39:10.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2026-02-25T16:39:10.325 INFO:tasks.workunit.client.0.trial129.stderr:osd.5 is already out. 2026-02-25T16:39:10.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2026-02-25T16:39:10.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2026-02-25T16:39:10.631 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 is already out. 2026-02-25T16:39:10.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2026-02-25T16:39:10.640 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:39:10.640 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:39:10.640 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:39:10.640 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:39:10.641 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:39:10.641 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:39:10.641 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:39:10.641 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:39:10.642 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:39:10.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:39:10.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:39:10.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:39:10.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:39:10.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:39:10.724 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:39:10.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:39:10.948 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:39:10.948 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:39:10.948 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:10.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:39:11.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836517 2026-02-25T16:39:11.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836517 2026-02-25T16:39:11.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517' 2026-02-25T16:39:11.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:39:11.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705702 2026-02-25T16:39:11.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705702 2026-02-25T16:39:11.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702' 2026-02-25T16:39:11.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:39:11.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574884 2026-02-25T16:39:11.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574884 2026-02-25T16:39:11.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884' 2026-02-25T16:39:11.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.141 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:39:11.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776153 2026-02-25T16:39:11.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776153 2026-02-25T16:39:11.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153' 2026-02-25T16:39:11.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:39:11.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247843 2026-02-25T16:39:11.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247843 2026-02-25T16:39:11.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153 4-98784247843' 2026-02-25T16:39:11.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.273 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:39:11.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579925 2026-02-25T16:39:11.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579925 2026-02-25T16:39:11.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153 4-98784247843 5-317827579925' 2026-02-25T16:39:11.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.336 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:39:11.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220177 2026-02-25T16:39:11.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220177 2026-02-25T16:39:11.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153 4-98784247843 5-317827579925 6-365072220177' 2026-02-25T16:39:11.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:39:11.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855394 2026-02-25T16:39:11.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855394 2026-02-25T16:39:11.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153 4-98784247843 5-317827579925 6-365072220177 7-150323855394' 2026-02-25T16:39:11.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.464 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:39:11.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724578 2026-02-25T16:39:11.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724578 2026-02-25T16:39:11.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153 4-98784247843 5-317827579925 6-365072220177 7-150323855394 8-167503724578' 2026-02-25T16:39:11.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.527 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:39:11.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023950 2026-02-25T16:39:11.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023950 2026-02-25T16:39:11.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153 4-98784247843 5-317827579925 6-365072220177 7-150323855394 8-167503724578 9-390842023950' 2026-02-25T16:39:11.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:11.589 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:39:11.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462945 2026-02-25T16:39:11.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462945 2026-02-25T16:39:11.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836517 1-38654705702 2-55834574884 3-292057776153 4-98784247843 5-317827579925 6-365072220177 7-150323855394 8-167503724578 9-390842023950 10-201863462945' 2026-02-25T16:39:11.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:11.654 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836517 2026-02-25T16:39:11.654 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:11.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:39:11.655 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836517 2026-02-25T16:39:11.655 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:11.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836517 2026-02-25T16:39:11.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2026-02-25T16:39:11.656 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 21474836517 2026-02-25T16:39:11.657 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:39:11.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836515 -lt 21474836517 2026-02-25T16:39:11.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:39:12.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:39:12.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:39:13.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836517 -lt 21474836517 2026-02-25T16:39:13.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:13.110 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705702 2026-02-25T16:39:13.110 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:13.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:39:13.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705702 2026-02-25T16:39:13.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:13.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705702 2026-02-25T16:39:13.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705702' 2026-02-25T16:39:13.113 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 38654705702 2026-02-25T16:39:13.113 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:39:13.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705702 -lt 38654705702 2026-02-25T16:39:13.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:13.340 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574884 2026-02-25T16:39:13.340 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:13.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:39:13.341 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574884 2026-02-25T16:39:13.341 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:13.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574884 2026-02-25T16:39:13.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574884' 2026-02-25T16:39:13.343 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 55834574884 2026-02-25T16:39:13.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:39:13.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574884 -lt 55834574884 2026-02-25T16:39:13.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:13.571 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-292057776153 2026-02-25T16:39:13.571 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:13.572 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:39:13.572 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776153 2026-02-25T16:39:13.572 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:13.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776153 2026-02-25T16:39:13.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 292057776153' 2026-02-25T16:39:13.574 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 292057776153 2026-02-25T16:39:13.574 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:39:13.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776153 -lt 292057776153 2026-02-25T16:39:13.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:13.801 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247843 2026-02-25T16:39:13.801 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:13.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:39:13.802 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247843 2026-02-25T16:39:13.802 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:13.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247843 2026-02-25T16:39:13.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247843' 2026-02-25T16:39:13.804 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 98784247843 2026-02-25T16:39:13.804 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:39:14.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247843 -lt 98784247843 2026-02-25T16:39:14.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:14.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-317827579925 2026-02-25T16:39:14.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:14.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:39:14.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-317827579925 2026-02-25T16:39:14.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:14.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=317827579925 2026-02-25T16:39:14.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 317827579925' 2026-02-25T16:39:14.037 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 317827579925 2026-02-25T16:39:14.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:39:14.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 317827579925 -lt 317827579925 2026-02-25T16:39:14.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:14.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-365072220177 2026-02-25T16:39:14.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:14.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:39:14.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-365072220177 2026-02-25T16:39:14.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:14.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=365072220177 2026-02-25T16:39:14.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 365072220177' 2026-02-25T16:39:14.269 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 365072220177 2026-02-25T16:39:14.270 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:39:14.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 365072220177 -lt 365072220177 2026-02-25T16:39:14.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:14.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855394 2026-02-25T16:39:14.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:14.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:39:14.500 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855394 2026-02-25T16:39:14.500 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:14.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855394 2026-02-25T16:39:14.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855394' 2026-02-25T16:39:14.501 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.7 seq 150323855394 2026-02-25T16:39:14.501 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:39:14.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855395 -lt 150323855394 2026-02-25T16:39:14.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:14.724 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724578 2026-02-25T16:39:14.724 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:14.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:39:14.727 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724578 2026-02-25T16:39:14.727 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:14.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724578 2026-02-25T16:39:14.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724578' 2026-02-25T16:39:14.728 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.8 seq 167503724578 2026-02-25T16:39:14.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:39:14.959 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724579 -lt 167503724578 2026-02-25T16:39:14.960 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:14.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-390842023950 2026-02-25T16:39:14.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:14.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:39:14.962 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-390842023950 2026-02-25T16:39:14.962 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=390842023950 2026-02-25T16:39:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 390842023950' 2026-02-25T16:39:14.963 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.9 seq 390842023950 2026-02-25T16:39:14.964 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:39:15.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 390842023950 -lt 390842023950 2026-02-25T16:39:15.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:15.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462945 2026-02-25T16:39:15.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:15.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:39:15.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462945 2026-02-25T16:39:15.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:15.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462945 2026-02-25T16:39:15.195 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.10 seq 201863462945 2026-02-25T16:39:15.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462945' 2026-02-25T16:39:15.195 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:39:15.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462945 -lt 201863462945 2026-02-25T16:39:15.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:39:15.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:39:15.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:39:15.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T16:39:15.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:39:15.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:39:15.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:39:15.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:39:15.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:39:15.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:39:15.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:39:15.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T16:39:15.958 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:39:15.958 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:39:15.958 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:39:16.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T16:39:16.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:39:16.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:39:16.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2026-02-25T16:39:16.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T16:39:16.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:39:16.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:16.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2026-02-25T16:39:16.270 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T16:39:16.270 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:16.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-25T16:39:16.500 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:16.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2026-02-25T16:39:16.502 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T16:39:16.502 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:16.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-25T16:39:16.730 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:39:16.730 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:16.730 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:16.730 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:16.731 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:39:16.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 7 9 4 3 10 2026-02-25T16:39:16.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:16.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:16.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2026-02-25T16:39:16.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2026-02-25T16:39:17.029 INFO:tasks.workunit.client.0.trial129.stderr:osd.5 is already in. 2026-02-25T16:39:17.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2026-02-25T16:39:17.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2026-02-25T16:39:17.322 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 is already in. 2026-02-25T16:39:17.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2026-02-25T16:39:17.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:39:17.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:39:17.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:39:17.331 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:39:17.331 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:39:17.331 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:39:17.331 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:39:17.331 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:39:17.331 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:39:17.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:39:17.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:39:17.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:39:17.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:39:17.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:39:17.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:39:17.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:39:17.645 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:39:17.645 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:8 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:17.646 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:39:17.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836520 2026-02-25T16:39:17.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836520 2026-02-25T16:39:17.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520' 2026-02-25T16:39:17.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:17.709 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:39:17.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705705 2026-02-25T16:39:17.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705705 2026-02-25T16:39:17.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705' 2026-02-25T16:39:17.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:17.771 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:39:17.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574887 2026-02-25T16:39:17.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574887 2026-02-25T16:39:17.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887' 2026-02-25T16:39:17.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:17.834 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:39:17.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776156 2026-02-25T16:39:17.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776156 2026-02-25T16:39:17.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156' 2026-02-25T16:39:17.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:17.898 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:39:17.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247846 2026-02-25T16:39:17.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247846 2026-02-25T16:39:17.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156 4-98784247846' 2026-02-25T16:39:17.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:17.965 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:39:18.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=317827579929 2026-02-25T16:39:18.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 317827579929 2026-02-25T16:39:18.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156 4-98784247846 5-317827579929' 2026-02-25T16:39:18.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:18.028 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:39:18.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220181 2026-02-25T16:39:18.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220181 2026-02-25T16:39:18.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156 4-98784247846 5-317827579929 6-365072220181' 2026-02-25T16:39:18.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:18.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:39:18.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855398 2026-02-25T16:39:18.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855398 2026-02-25T16:39:18.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156 4-98784247846 5-317827579929 6-365072220181 7-150323855398' 2026-02-25T16:39:18.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:18.153 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:39:18.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724581 2026-02-25T16:39:18.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724581 2026-02-25T16:39:18.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156 4-98784247846 5-317827579929 6-365072220181 7-150323855398 8-167503724581' 2026-02-25T16:39:18.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:18.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-25T16:39:18.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023953 2026-02-25T16:39:18.281 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023953 2026-02-25T16:39:18.281 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156 4-98784247846 5-317827579929 6-365072220181 7-150323855398 8-167503724581 9-390842023953' 2026-02-25T16:39:18.281 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:39:18.281 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.10 flush_pg_stats 2026-02-25T16:39:18.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462948 2026-02-25T16:39:18.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462948 2026-02-25T16:39:18.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836520 1-38654705705 2-55834574887 3-292057776156 4-98784247846 5-317827579929 6-365072220181 7-150323855398 8-167503724581 9-390842023953 10-201863462948' 2026-02-25T16:39:18.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:18.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836520 2026-02-25T16:39:18.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:18.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:39:18.345 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836520 2026-02-25T16:39:18.346 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:18.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836520 2026-02-25T16:39:18.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2026-02-25T16:39:18.347 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 21474836520 2026-02-25T16:39:18.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:39:18.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836518 -lt 21474836520 2026-02-25T16:39:18.575 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:39:19.575 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:39:19.575 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:39:19.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836520 -lt 21474836520 2026-02-25T16:39:19.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:19.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705705 2026-02-25T16:39:19.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:19.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:39:19.808 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705705 2026-02-25T16:39:19.808 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:19.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705705 2026-02-25T16:39:19.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705705' 2026-02-25T16:39:19.809 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 38654705705 2026-02-25T16:39:19.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:39:20.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705705 -lt 38654705705 2026-02-25T16:39:20.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:20.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574887 2026-02-25T16:39:20.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:20.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:39:20.035 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574887 2026-02-25T16:39:20.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:20.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574887 2026-02-25T16:39:20.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574887' 2026-02-25T16:39:20.037 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 55834574887 2026-02-25T16:39:20.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:39:20.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574887 -lt 55834574887 2026-02-25T16:39:20.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:20.262 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-292057776156 2026-02-25T16:39:20.262 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:20.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:39:20.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-292057776156 2026-02-25T16:39:20.264 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:20.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776156 2026-02-25T16:39:20.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 292057776156' 2026-02-25T16:39:20.265 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 292057776156 2026-02-25T16:39:20.265 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:39:20.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776156 -lt 292057776156 2026-02-25T16:39:20.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:20.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-98784247846 2026-02-25T16:39:20.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:20.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:39:20.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-98784247846 2026-02-25T16:39:20.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:20.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=98784247846 2026-02-25T16:39:20.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 98784247846' 2026-02-25T16:39:20.497 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 98784247846 2026-02-25T16:39:20.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:39:20.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 98784247847 -lt 98784247846 2026-02-25T16:39:20.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:20.725 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-317827579929 2026-02-25T16:39:20.725 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:20.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:39:20.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-317827579929 2026-02-25T16:39:20.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:20.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=317827579929 2026-02-25T16:39:20.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 317827579929' 2026-02-25T16:39:20.727 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 317827579929 2026-02-25T16:39:20.728 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:39:20.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 317827579929 -lt 317827579929 2026-02-25T16:39:20.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:20.951 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-365072220181 2026-02-25T16:39:20.951 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:20.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:39:20.952 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-365072220181 2026-02-25T16:39:20.952 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:20.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=365072220181 2026-02-25T16:39:20.953 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 365072220181 2026-02-25T16:39:20.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 365072220181' 2026-02-25T16:39:20.954 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:39:21.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 365072220181 -lt 365072220181 2026-02-25T16:39:21.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:21.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-150323855398 2026-02-25T16:39:21.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:21.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:39:21.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-150323855398 2026-02-25T16:39:21.178 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:21.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855398 2026-02-25T16:39:21.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 150323855398' 2026-02-25T16:39:21.179 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.7 seq 150323855398 2026-02-25T16:39:21.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:39:21.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855398 -lt 150323855398 2026-02-25T16:39:21.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:21.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-167503724581 2026-02-25T16:39:21.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:21.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:39:21.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-167503724581 2026-02-25T16:39:21.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:21.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724581 2026-02-25T16:39:21.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 167503724581' 2026-02-25T16:39:21.410 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.8 seq 167503724581 2026-02-25T16:39:21.410 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:39:21.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724582 -lt 167503724581 2026-02-25T16:39:21.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:21.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 9-390842023953 2026-02-25T16:39:21.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:21.637 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=9 2026-02-25T16:39:21.637 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-390842023953 2026-02-25T16:39:21.637 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:21.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=390842023953 2026-02-25T16:39:21.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.9 seq 390842023953' 2026-02-25T16:39:21.639 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.9 seq 390842023953 2026-02-25T16:39:21.639 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-25T16:39:21.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 390842023953 -lt 390842023953 2026-02-25T16:39:21.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:39:21.866 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 10-201863462948 2026-02-25T16:39:21.866 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:39:21.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=10 2026-02-25T16:39:21.868 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 10-201863462948 2026-02-25T16:39:21.868 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:39:21.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=201863462948 2026-02-25T16:39:21.869 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.10 seq 201863462948 2026-02-25T16:39:21.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.10 seq 201863462948' 2026-02-25T16:39:21.869 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-25T16:39:22.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 201863462949 -lt 201863462948 2026-02-25T16:39:22.096 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:39:22.096 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:39:22.096 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:39:22.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 28 == 0 2026-02-25T16:39:22.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:39:22.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:39:22.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:39:22.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:39:22.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:39:22.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:39:22.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:39:22.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=28 2026-02-25T16:39:22.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:39:22.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:39:22.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:39:22.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 = 28 2026-02-25T16:39:22.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:39:22.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:39:22.945 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-25T16:39:22.945 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:39:22.945 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:22.945 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-25T16:39:22.945 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:23.172 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T16:39:23.172 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:39:23.172 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:23.172 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:39:23.172 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:23.172 INFO:tasks.workunit.client.0.trial129.stderr:10' 2026-02-25T16:39:23.173 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 1 9 4 3 10 2026-02-25T16:39:23.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2026-02-25T16:39:23.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:39:23.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2026-02-25T16:39:23.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-jerasure 2026-02-25T16:39:23.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:39:23.417 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:39:23.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2026-02-25T16:39:23.702 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile profile-jerasure does not exist 2026-02-25T16:39:23.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T16:39:23.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2026-02-25T16:39:23.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2026-02-25T16:39:23.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2026-02-25T16:39:23.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2026-02-25T16:39:23.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2026-02-25T16:39:23.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2026-02-25T16:39:23.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2026-02-25T16:39:24.376 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-lrc-a' already exists 2026-02-25T16:39:24.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:39:25.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2026-02-25T16:39:25.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T16:39:25.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T16:39:25.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T16:39:25.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:25.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:25.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T16:39:25.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2026-02-25T16:39:25.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2026-02-25T16:39:25.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:25.425 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2026-02-25T16:39:25.425 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 3 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2026-02-25T16:39:25.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2026-02-25T16:39:25.901 INFO:tasks.workunit.client.0.trial129.stderr:osd.3 is already out. 2026-02-25T16:39:25.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T16:39:30.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2026-02-25T16:39:30.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc-a 2026-02-25T16:39:30.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:30.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2026-02-25T16:39:30.911 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2026-02-25T16:39:30.911 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:31.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=5 2026-02-25T16:39:31.145 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:31.145 INFO:tasks.workunit.client.0.trial129.stderr:7' 2026-02-25T16:39:31.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 5 9 7 2026-02-25T16:39:31.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:31.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:31.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2026-02-25T16:39:31.393 INFO:tasks.workunit.client.0.trial129.stderr:osd.3 is already in. 2026-02-25T16:39:31.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:39:31.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2026-02-25T16:39:31.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc-a 2026-02-25T16:39:31.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2026-02-25T16:39:31.695 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-lrc-a' does not exist 2026-02-25T16:39:31.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2026-02-25T16:39:31.978 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile profile-lrc-a does not exist 2026-02-25T16:39:31.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T16:39:31.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2026-02-25T16:39:31.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2026-02-25T16:39:31.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2026-02-25T16:39:31.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2026-02-25T16:39:31.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2026-02-25T16:39:32.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2026-02-25T16:39:32.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2026-02-25T16:39:32.646 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-lrc' already exists 2026-02-25T16:39:32.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:39:33.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2026-02-25T16:39:33.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T16:39:33.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2026-02-25T16:39:33.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T16:39:33.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T16:39:33.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T16:39:33.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:33.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:33.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T16:39:33.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2026-02-25T16:39:33.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2026-02-25T16:39:33.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:33.694 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2026-02-25T16:39:33.694 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:33.924 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2026-02-25T16:39:33.924 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:10 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:7' 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 7 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T16:39:33.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2026-02-25T16:39:33.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2026-02-25T16:39:34.175 INFO:tasks.workunit.client.0.trial129.stderr:osd.7 is already out. 2026-02-25T16:39:34.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T16:39:39.184 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2026-02-25T16:39:39.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-lrc 2026-02-25T16:39:39.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:39.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2026-02-25T16:39:39.185 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-lrc SOMETHING 2026-02-25T16:39:39.185 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=8 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:10 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:4' 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 8 6 0 10 5 3 9 4 2026-02-25T16:39:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:39.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:39.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2026-02-25T16:39:39.663 INFO:tasks.workunit.client.0.trial129.stderr:osd.7 is already in. 2026-02-25T16:39:39.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:39:39.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2026-02-25T16:39:39.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-lrc 2026-02-25T16:39:39.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2026-02-25T16:39:39.960 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-lrc' does not exist 2026-02-25T16:39:39.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2026-02-25T16:39:40.252 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile profile-lrc does not exist 2026-02-25T16:39:40.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-25T16:39:40.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2026-02-25T16:39:40.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2026-02-25T16:39:40.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2026-02-25T16:39:40.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2026-02-25T16:39:40.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2026-02-25T16:39:40.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2026-02-25T16:39:40.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2026-02-25T16:39:40.928 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-shec' already exists 2026-02-25T16:39:40.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-25T16:39:41.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-25T16:39:41.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:41.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:41.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-25T16:39:41.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2026-02-25T16:39:41.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2026-02-25T16:39:41.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:41.976 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2026-02-25T16:39:41.977 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr:1' 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 1 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2026-02-25T16:39:42.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2026-02-25T16:39:42.458 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 is already out. 2026-02-25T16:39:42.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-25T16:39:47.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2026-02-25T16:39:47.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-shec 2026-02-25T16:39:47.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-25T16:39:47.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2026-02-25T16:39:47.468 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-shec SOMETHING 2026-02-25T16:39:47.468 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:39:47.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=7 2026-02-25T16:39:47.694 INFO:tasks.workunit.client.0.trial129.stderr:9 2026-02-25T16:39:47.694 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:39:47.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 7 9 6 2026-02-25T16:39:47.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2026-02-25T16:39:47.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-25T16:39:47.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2026-02-25T16:39:47.945 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 is already in. 2026-02-25T16:39:47.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-25T16:39:47.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2026-02-25T16:39:47.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=pool-shec 2026-02-25T16:39:47.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2026-02-25T16:39:48.243 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-shec' does not exist 2026-02-25T16:39:48.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2026-02-25T16:39:48.530 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile profile-shec does not exist 2026-02-25T16:39:48.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2026-02-25T16:39:48.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=ecpool 2026-02-25T16:39:48.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2026-02-25T16:39:48.806 INFO:tasks.workunit.client.0.trial129.stderr:pool 'ecpool' does not exist 2026-02-25T16:39:48.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2026-02-25T16:39:48.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2026-02-25T16:39:48.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:39:48.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2026-02-25T16:39:48.815 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:39:48.815 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:39:48.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:39:48.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:39:48.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:39:48.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:39:48.927 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:39:48.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:39:48.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:39:48.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:39:48.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:39:48.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:39:48.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:39:48.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:39:48.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:39:48.931 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:39:48.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:39:48.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:39:48.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2026-02-25T16:39:48.989 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:39:48.990 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:48.990 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:39:48.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.68264 2026-02-25T16:39:48.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:39:48.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:39:48.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T16:39:48.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/test-erasure-code 0 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-code 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-code KILL 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:39:48.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:39:48.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:39:48.993 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:39:48.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:39:48.994 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:39:48.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:39:48.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:39:48.995 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:39:48.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:39:48.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:39:48.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:39:48.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:39:48.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:39:48.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T16:39:48.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-code 2026-02-25T16:39:48.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:39:48.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:48.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.68264 2026-02-25T16:39:49.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.68264 2026-02-25T16:39:49.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:39:49.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:39:49.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T16:39:49.001 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T16:39:49.001 DEBUG:teuthology.orchestra.run.trial129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T16:39:49.067 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2026-02-25T16:39:49.067 DEBUG:teuthology.orchestra.run.trial129:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2026-02-25T16:39:49.128 INFO:tasks.workunit.client.0.trial129.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/test-erasure-eio 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2026-02-25T16:39:49.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2026-02-25T16:39:49.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2026-02-25T16:39:49.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2026-02-25T16:39:49.132 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2026-02-25T16:39:49.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none ' 2026-02-25T16:39:49.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2026-02-25T16:39:49.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2026-02-25T16:39:49.133 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2026-02-25T16:39:49.133 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_ec_object_attr_read_error 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_ec_recovery_multiple_errors 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_ec_recovery_multiple_objects 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_ec_recovery_multiple_objects_eio 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_ec_recovery_unfound 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_ec_single_recovery_error 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_get_bad_size_shard_0 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_get_bad_size_shard_1 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_get_subread_eio_shard_0 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_get_subread_eio_shard_1 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_get_subread_missing 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:39:49.135 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:39:49.136 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:39:49.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:39:49.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:39:49.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:39:49.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:39:49.137 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:39:49.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:39:49.138 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:39:49.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:39:49.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:39:49.139 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:39:49.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:39:49.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:39:49.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:39:49.140 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:39:49.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:39:49.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:39:49.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:39:49.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:39:49.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:49.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:49.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:39:49.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:39:49.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:39:49.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:39:49.145 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:39:49.145 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:49.145 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:49.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:39:49.146 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:39:49.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:49.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:39:49.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:39:49.189 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:39:49.190 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:39:49.190 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:49.190 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:49.190 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:39:49.190 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:39:49.190 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:49.236 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:49.237 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:39:49.237 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:39:49.237 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:39:49.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:39:49.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:39:49.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:39:49.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:39:49.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:39:49.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:39:49.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:39:49.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:39:49.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:39:49.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:39:49.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:39:49.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:49.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:49.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:49.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:39:49.381 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:39:49.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:39:49.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:39:49.491 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:39:49.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:39:50.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:39:50.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:39:50.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:39:50.500 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:39:50.500 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:50.500 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:50.500 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:39:50.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:39:50.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:39:50.538 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:39:50.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:39:49.180+0000 7fdaa6262c00 0 load: jerasure load: lrc 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: local objname=myobject 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: local lastobj=300 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:535: TEST_ec_backfill_unfound: local testobj=obj250 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:39:50.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: setup_osds 5 2026-02-25T16:39:50.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2026-02-25T16:39:50.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:39:50.544 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2026-02-25T16:39:50.544 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2026-02-25T16:39:50.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:50.546 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:39:50.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:39:50.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c32ffce9-935a-4b56-bc28-2d1c0007889e 2026-02-25T16:39:50.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c32ffce9-935a-4b56-bc28-2d1c0007889e' 2026-02-25T16:39:50.548 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 c32ffce9-935a-4b56-bc28-2d1c0007889e 2026-02-25T16:39:50.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:39:50.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWJZ9pIMJDIRAAYc7iaXtmHf7kpuALL145Gw== 2026-02-25T16:39:50.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWJZ9pIMJDIRAAYc7iaXtmHf7kpuALL145Gw=="}' 2026-02-25T16:39:50.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c32ffce9-935a-4b56-bc28-2d1c0007889e -i td/test-erasure-eio/0/new.json 2026-02-25T16:39:50.663 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:39:50.669 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:39:50.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWJZ9pIMJDIRAAYc7iaXtmHf7kpuALL145Gw== --osd-uuid c32ffce9-935a-4b56-bc28-2d1c0007889e 2026-02-25T16:39:50.688 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:50.686+0000 7fc8dfc62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:50.689 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:50.688+0000 7fc8dfc62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:50.690 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:50.689+0000 7fc8dfc62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:50.690 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:50.689+0000 7fc8dfc62780 -1 bdev(0x33bec00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:39:50.690 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:50.689+0000 7fc8dfc62780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:39:51.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:39:51.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:39:51.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:39:51.076 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:39:51.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:39:51.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:39:51.180 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:39:51.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:39:51.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:39:51.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:39:51.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:39:51.196 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:51.194+0000 7f2984661780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:51.197 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:51.196+0000 7f2984661780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:51.198 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:51.197+0000 7f2984661780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:51.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:39:51.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:39:51.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:39:51.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:39:51.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:39:51.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:39:51.279 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:39:51.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:39:51.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:39:51.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:39:51.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:39:51.404 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:51.403+0000 7f2984661780 -1 Falling back to public interface 2026-02-25T16:39:51.690 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:51.688+0000 7f2984661780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:39:52.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:39:52.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:39:52.378 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:39:52.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:39:52.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:39:52.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:39:52.605 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/869166476,v1:127.0.0.1:6803/869166476] [v2:127.0.0.1:6804/869166476,v1:127.0.0.1:6805/869166476] exists,up c32ffce9-935a-4b56-bc28-2d1c0007889e 2026-02-25T16:39:52.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:39:52.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:39:52.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:39:52.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:39:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:52.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:39:52.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=daf0e2c2-53df-4956-a3ca-79588cff32f9 2026-02-25T16:39:52.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 daf0e2c2-53df-4956-a3ca-79588cff32f9' 2026-02-25T16:39:52.613 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 daf0e2c2-53df-4956-a3ca-79588cff32f9 2026-02-25T16:39:52.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:39:52.619 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDYJZ9pjQPYJBAAQyQ6+gibDCVsqrzw0FVRxg== 2026-02-25T16:39:52.619 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDYJZ9pjQPYJBAAQyQ6+gibDCVsqrzw0FVRxg=="}' 2026-02-25T16:39:52.619 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new daf0e2c2-53df-4956-a3ca-79588cff32f9 -i td/test-erasure-eio/1/new.json 2026-02-25T16:39:52.854 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:39:52.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:39:52.863 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYJZ9pjQPYJBAAQyQ6+gibDCVsqrzw0FVRxg== --osd-uuid daf0e2c2-53df-4956-a3ca-79588cff32f9 2026-02-25T16:39:52.881 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:52.879+0000 7f1fc5a2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:52.882 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:52.880+0000 7f1fc5a2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:52.883 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:52.881+0000 7f1fc5a2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:52.883 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:52.881+0000 7f1fc5a2d780 -1 bdev(0x2e3ac00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:39:52.884 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:52.881+0000 7f1fc5a2d780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:39:53.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:39:53.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:39:53.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:39:53.224 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:39:53.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:39:53.543 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:39:53.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:39:53.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:39:53.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:39:53.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:39:53.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:39:53.560 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:53.559+0000 7fbdb9e32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:53.562 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:53.560+0000 7fbdb9e32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:53.563 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:53.561+0000 7fbdb9e32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:53.722 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:53.720+0000 7fbdb9e32780 -1 Falling back to public interface 2026-02-25T16:39:53.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:39:53.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:39:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:39:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:39:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:39:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:39:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:39:53.774 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:39:53.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:39:53.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:39:53.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:39:54.050 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:54.048+0000 7fbdb9e32780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:39:55.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:39:55.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:39:55.001 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:39:55.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:39:55.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:39:55.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:39:55.244 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1538748658,v1:127.0.0.1:6811/1538748658] [v2:127.0.0.1:6812/1538748658,v1:127.0.0.1:6813/1538748658] exists,up daf0e2c2-53df-4956-a3ca-79588cff32f9 2026-02-25T16:39:55.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:39:55.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:39:55.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:39:55.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:39:55.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:39:55.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:39:55.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:39:55.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:39:55.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:39:55.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:39:55.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:39:55.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:39:55.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:39:55.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:39:55.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:39:55.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:39:55.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:39:55.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:39:55.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:55.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=86b39497-c715-423c-8c40-ddaed7e93a46 2026-02-25T16:39:55.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 86b39497-c715-423c-8c40-ddaed7e93a46' 2026-02-25T16:39:55.257 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 86b39497-c715-423c-8c40-ddaed7e93a46 2026-02-25T16:39:55.257 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:39:55.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbJZ9p8o5TDxAAk1vvS4qrWozjsStzkhYBJQ== 2026-02-25T16:39:55.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbJZ9p8o5TDxAAk1vvS4qrWozjsStzkhYBJQ=="}' 2026-02-25T16:39:55.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 86b39497-c715-423c-8c40-ddaed7e93a46 -i td/test-erasure-eio/2/new.json 2026-02-25T16:39:55.489 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:39:55.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:39:55.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbJZ9p8o5TDxAAk1vvS4qrWozjsStzkhYBJQ== --osd-uuid 86b39497-c715-423c-8c40-ddaed7e93a46 2026-02-25T16:39:55.515 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:55.514+0000 7f14dfa4d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:55.517 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:55.515+0000 7f14dfa4d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:55.518 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:55.516+0000 7f14dfa4d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:55.518 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:55.517+0000 7f14dfa4d780 -1 bdev(0x408cc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:39:55.518 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:55.517+0000 7f14dfa4d780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:39:55.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:39:55.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:39:55.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:39:55.869 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:39:55.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:39:56.187 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:39:56.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:39:56.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:39:56.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:39:56.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:39:56.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:39:56.204 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:56.202+0000 7f4f06e37780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:56.205 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:56.204+0000 7f4f06e37780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:56.206 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:56.204+0000 7f4f06e37780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:56.362 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:56.360+0000 7f4f06e37780 -1 Falling back to public interface 2026-02-25T16:39:56.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:39:56.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:39:56.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:39:56.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:39:56.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:39:56.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:39:56.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:39:56.418 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:39:56.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:39:56.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:39:56.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:39:56.704 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:56.702+0000 7f4f06e37780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:39:57.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:39:57.645 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:39:57.646 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:39:57.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:39:57.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:39:57.646 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:39:57.878 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3345151360,v1:127.0.0.1:6819/3345151360] [v2:127.0.0.1:6820/3345151360,v1:127.0.0.1:6821/3345151360] exists,up 86b39497-c715-423c-8c40-ddaed7e93a46 2026-02-25T16:39:57.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:39:57.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:39:57.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:39:57.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:39:57.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:39:57.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7608f4bb-3a54-4bd5-bcd8-5e1927f86358 2026-02-25T16:39:57.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 7608f4bb-3a54-4bd5-bcd8-5e1927f86358' 2026-02-25T16:39:57.885 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 7608f4bb-3a54-4bd5-bcd8-5e1927f86358 2026-02-25T16:39:57.885 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:39:57.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDdJZ9pc/tENRAAxCwLrr/ZrYhEIj5V1QkiDA== 2026-02-25T16:39:57.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDdJZ9pc/tENRAAxCwLrr/ZrYhEIj5V1QkiDA=="}' 2026-02-25T16:39:57.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7608f4bb-3a54-4bd5-bcd8-5e1927f86358 -i td/test-erasure-eio/3/new.json 2026-02-25T16:39:58.125 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:39:58.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:39:58.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdJZ9pc/tENRAAxCwLrr/ZrYhEIj5V1QkiDA== --osd-uuid 7608f4bb-3a54-4bd5-bcd8-5e1927f86358 2026-02-25T16:39:58.151 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.150+0000 7f86d0a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:58.153 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.151+0000 7f86d0a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:58.154 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.152+0000 7f86d0a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:58.154 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.152+0000 7f86d0a0d780 -1 bdev(0x3320c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:39:58.155 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.152+0000 7f86d0a0d780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:39:58.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:39:58.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:39:58.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:39:58.522 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:39:58.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:39:58.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:39:58.839 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:39:58.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:39:58.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:39:58.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:39:58.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:39:58.855 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.854+0000 7fbea9e87780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:58.857 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.855+0000 7fbea9e87780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:58.858 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.856+0000 7fbea9e87780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:39:59.001 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:58.999+0000 7fbea9e87780 -1 Falling back to public interface 2026-02-25T16:39:59.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:39:59.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:39:59.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:39:59.361 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:39:59.359+0000 7fbea9e87780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:40:00.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:40:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:00.294 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:40:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:40:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:40:00.527 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2805497813,v1:127.0.0.1:6827/2805497813] [v2:127.0.0.1:6828/2805497813,v1:127.0.0.1:6829/2805497813] exists,up 7608f4bb-3a54-4bd5-bcd8-5e1927f86358 2026-02-25T16:40:00.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:40:00.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:40:00.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:40:00.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:40:00.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T16:40:00.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:40:00.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:40:00.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:40:00.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:40:00.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=83fc66de-a09b-4c27-8a9f-bee5cd421467 2026-02-25T16:40:00.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 83fc66de-a09b-4c27-8a9f-bee5cd421467' 2026-02-25T16:40:00.536 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 83fc66de-a09b-4c27-8a9f-bee5cd421467 2026-02-25T16:40:00.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:40:00.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDgJZ9puPA0IBAA5mkdE44vN6Wq67l4qfakNA== 2026-02-25T16:40:00.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDgJZ9puPA0IBAA5mkdE44vN6Wq67l4qfakNA=="}' 2026-02-25T16:40:00.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 83fc66de-a09b-4c27-8a9f-bee5cd421467 -i td/test-erasure-eio/4/new.json 2026-02-25T16:40:00.773 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:40:00.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T16:40:00.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgJZ9puPA0IBAA5mkdE44vN6Wq67l4qfakNA== --osd-uuid 83fc66de-a09b-4c27-8a9f-bee5cd421467 2026-02-25T16:40:00.800 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:00.798+0000 7fcf45a2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:00.801 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:00.800+0000 7fcf45a2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:00.802 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:00.800+0000 7fcf45a2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:00.802 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:00.801+0000 7fcf45a2e780 -1 bdev(0x3546c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:40:00.802 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:00.801+0000 7fcf45a2e780 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T16:40:01.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T16:40:01.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:40:01.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:40:01.156 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:40:01.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:40:01.474 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:40:01.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:40:01.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:40:01.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:40:01.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:40:01.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:40:01.491 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:01.489+0000 7fc2f8e10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:01.492 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:01.491+0000 7fc2f8e10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:01.493 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:01.492+0000 7fc2f8e10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:01.654 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:01.652+0000 7fc2f8e10780 -1 Falling back to public interface 2026-02-25T16:40:01.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:40:01.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:40:01.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:40:01.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:40:01.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:40:01.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:01.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:40:01.704 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:40:01.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:01.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:40:01.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:40:01.986 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:01.984+0000 7fc2f8e10780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:40:02.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:40:02.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:02.932 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:40:02.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:40:02.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:02.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:40:03.165 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 35 up_thru 36 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1699498025,v1:127.0.0.1:6835/1699498025] [v2:127.0.0.1:6836/1699498025,v1:127.0.0.1:6837/1699498025] exists,up 83fc66de-a09b-4c27-8a9f-bee5cd421467 2026-02-25T16:40:03.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:40:03.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:40:03.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:40:03.166 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:40:03.166 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:40:03.166 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:40:03.167 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:03.167 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:03.167 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:03.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:40:03.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:40:03.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:40:03.212 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:40:03.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:40:03.218 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:39:51.405+0000 7f2984661780 0 load: jerasure load: lrc 2026-02-25T16:40:03.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:540: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:40:03.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:542: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2026-02-25T16:40:03.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:543: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T16:40:03.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:40:03.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:40:03.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T16:40:03.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:40:03.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T16:40:03.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:40:03.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T16:40:03.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:40:03.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:40:03.879 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:40:03.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:40:04.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:40:04.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:40:04.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:40:04.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:40:04.889 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:40:04.890 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:40:04.890 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:40:04.891 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:40:04.891 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:40:04.891 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:40:04.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:40:04.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:40:04.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:40:04.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:40:04.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:40:04.974 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr:4' 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:05.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:40:05.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803782 2026-02-25T16:40:05.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803782 2026-02-25T16:40:05.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782' 2026-02-25T16:40:05.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:05.266 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:40:05.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607558 2026-02-25T16:40:05.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607558 2026-02-25T16:40:05.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-51539607558' 2026-02-25T16:40:05.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:05.333 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:40:05.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-25T16:40:05.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-25T16:40:05.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-51539607558 2-77309411332' 2026-02-25T16:40:05.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:05.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:40:05.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116995 2026-02-25T16:40:05.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116995 2026-02-25T16:40:05.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-51539607558 2-77309411332 3-115964116995' 2026-02-25T16:40:05.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:05.462 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:40:05.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855363 2026-02-25T16:40:05.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855363 2026-02-25T16:40:05.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-51539607558 2-77309411332 3-115964116995 4-150323855363' 2026-02-25T16:40:05.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:05.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803782 2026-02-25T16:40:05.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:05.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:40:05.528 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803782 2026-02-25T16:40:05.528 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:05.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803782 2026-02-25T16:40:05.529 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803782 2026-02-25T16:40:05.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803782' 2026-02-25T16:40:05.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:40:05.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803782 2026-02-25T16:40:05.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:05.756 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607558 2026-02-25T16:40:05.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:05.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:40:05.758 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607558 2026-02-25T16:40:05.758 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:05.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607558 2026-02-25T16:40:05.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607558' 2026-02-25T16:40:05.760 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607558 2026-02-25T16:40:05.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:40:05.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607558 -lt 51539607558 2026-02-25T16:40:05.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:05.985 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411332 2026-02-25T16:40:05.985 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:05.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:40:05.986 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411332 2026-02-25T16:40:05.986 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:05.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411332 2026-02-25T16:40:05.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-25T16:40:05.988 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411332 2026-02-25T16:40:05.988 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:40:06.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411330 -lt 77309411332 2026-02-25T16:40:06.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:40:07.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:40:07.214 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:40:07.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411332 -lt 77309411332 2026-02-25T16:40:07.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:07.442 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116995 2026-02-25T16:40:07.442 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:07.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:40:07.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116995 2026-02-25T16:40:07.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:07.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116995 2026-02-25T16:40:07.446 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116995' 2026-02-25T16:40:07.446 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116995 2026-02-25T16:40:07.446 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:40:07.671 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116996 -lt 115964116995 2026-02-25T16:40:07.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:07.672 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855363 2026-02-25T16:40:07.672 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:07.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:40:07.674 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855363 2026-02-25T16:40:07.674 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:07.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855363 2026-02-25T16:40:07.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855363' 2026-02-25T16:40:07.675 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855363 2026-02-25T16:40:07.675 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:40:07.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855363 -lt 150323855363 2026-02-25T16:40:07.899 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:40:07.899 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:40:07.899 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:40:08.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:40:08.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:40:08.430 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:40:08.430 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:40:08.430 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:40:08.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:40:08.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:40:08.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:40:08.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T16:40:08.954 INFO:tasks.workunit.client.0.trial129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:05.016502+0000 0'0 42:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T16:40:03.814729+0000 0'0 2026-02-25T16:40:03.814729+0000 0 0 periodic scrub scheduled @ 2026-02-27T02:38:36.168034+0000 0 0 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.017007+0000 0'0 42:20 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T21:41:31.039885+0000 0 0 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.016888+0000 0'0 41:97 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:22:19.353288+0000 0 0 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.419059+0000 0'0 42:20 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T20:17:11.594100+0000 0 0 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:39:57.853594+0000 0'0 42:69 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:18:14.101126+0000 0 0 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stdout: 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T16:40:08.955 INFO:tasks.workunit.client.0.trial129.stderr:dumped pgs 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:40:08.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:08.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2026-02-25T16:40:08.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2026-02-25T16:40:08.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2026-02-25T16:40:08.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:40:08.981 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T16:40:09.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: local primary=3 2026-02-25T16:40:09.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2026-02-25T16:40:09.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:40:09.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:40:09.215 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:40:09.215 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2026-02-25T16:40:09.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: local -a initial_osds 2026-02-25T16:40:09.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: local last_osd=2 2026-02-25T16:40:09.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:40:09.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:40:09.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:40:09.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:40:09.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:40:09.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:40:09.751 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:40:09.751 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: ceph osd down 2 2026-02-25T16:40:09.976 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already down. 2026-02-25T16:40:09.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: ceph osd out 2 2026-02-25T16:40:10.231 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already out. 2026-02-25T16:40:10.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T16:40:10.451 INFO:tasks.workunit.client.0.trial129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T16:40:10.451 INFO:tasks.workunit.client.0.trial129.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:05.016502+0000 0'0 42:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T16:40:03.814729+0000 0'0 2026-02-25T16:40:03.814729+0000 0 0 periodic scrub scheduled @ 2026-02-27T02:38:36.168034+0000 0 0 2026-02-25T16:40:10.451 INFO:tasks.workunit.client.0.trial129.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.017007+0000 0'0 42:20 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T21:41:31.039885+0000 0 0 2026-02-25T16:40:10.452 INFO:tasks.workunit.client.0.trial129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.016888+0000 0'0 42:99 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:22:19.353288+0000 0 0 2026-02-25T16:40:10.452 INFO:tasks.workunit.client.0.trial129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.419059+0000 0'0 42:20 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T20:17:11.594100+0000 0 0 2026-02-25T16:40:10.452 INFO:tasks.workunit.client.0.trial129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2026-02-25T16:40:09.600144+0000 0'0 44:77 [1,0] 1 [1,0] 1 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T23:42:34.519823+0000 0 0 2026-02-25T16:40:10.452 INFO:tasks.workunit.client.0.trial129.stdout: 2026-02-25T16:40:10.452 INFO:tasks.workunit.client.0.trial129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T16:40:10.452 INFO:tasks.workunit.client.0.trial129.stderr:dumped pgs 2026-02-25T16:40:10.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2026-02-25T16:40:10.460 INFO:tasks.workunit.client.0.trial129.stderr:4+0 records in 2026-02-25T16:40:10.460 INFO:tasks.workunit.client.0.trial129.stderr:4+0 records out 2026-02-25T16:40:10.460 INFO:tasks.workunit.client.0.trial129.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.2213e-05 s, 44.4 MB/s 2026-02-25T16:40:10.460 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: seq 1 300 2026-02-25T16:40:10.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.575 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.575 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:10.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:10.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.054 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.054 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.086 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.086 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.102 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.102 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.452 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.452 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:11.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:11.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.893 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.893 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:12.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:12.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.328 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.328 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.619 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.619 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.669 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.769 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.769 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.870 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.870 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.918 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:13.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:13.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.184 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.184 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.450 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.578 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.578 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.595 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.595 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.772 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.821 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.821 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.870 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.870 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:14.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:14.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.098 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.098 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:15.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2026-02-25T16:40:15.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2026-02-25T16:40:15.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:40:15.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:40:15.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:40:15.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:40:15.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj250 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj250 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2026-02-25T16:40:15.414 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:40:15.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:40:15.638 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:40:15.638 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:40:15.638 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:40:15.638 INFO:tasks.workunit.client.0.trial129.stderr:2147483647' 2026-02-25T16:40:15.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T16:40:15.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T16:40:15.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:40:15.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T16:40:15.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:40:15.639 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T16:40:15.640 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:15.641 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:15.642 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:15.642 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:40:15.642 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:40:15.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:40:15.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:40:15.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:40:15.691 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T16:40:15.691 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:40:15.691 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:40:15.691 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:15.691 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:15.691 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:15.692 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:40:15.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:40:15.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2026-02-25T16:40:15.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:565: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2026-02-25T16:40:15.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:40:15.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:40:15.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:40:15.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:40:15.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:40:15.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:40:15.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj250 2026-02-25T16:40:15.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:40:15.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:40:15.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:40:15.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T16:40:15.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:40:15.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj250 2026-02-25T16:40:15.744 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:40:15.744 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj250 2026-02-25T16:40:15.744 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj250 2026-02-25T16:40:15.744 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:2147483647' 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:40:15.971 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T16:40:15.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:40:15.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T16:40:15.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:40:15.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T16:40:15.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:40:15.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:40:15.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:40:15.973 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T16:40:15.973 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:40:15.973 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:40:15.973 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:15.974 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:15.974 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:15.974 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:40:15.974 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:40:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:40:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:40:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:40:16.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T16:40:16.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:40:16.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:40:16.021 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:16.021 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:16.021 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:16.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:40:16.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:40:16.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2026-02-25T16:40:16.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2026-02-25T16:40:16.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:40:16.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:40:16.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:40:16.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:40:16.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:40:16.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:40:16.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:40:16.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:40:16.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:40:16.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:40:16.075 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:40:16.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:40:16.076 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:40:16.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:40:16.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:40:16.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:40:16.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:40:16.085 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:16.083+0000 7f9b25e51780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:16.086 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:16.084+0000 7f9b25e51780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:16.087 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:16.085+0000 7f9b25e51780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:16.221 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:16.219+0000 7f9b25e51780 -1 Falling back to public interface 2026-02-25T16:40:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:40:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:40:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:40:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:40:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:40:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:16.305 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:40:16.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:40:16.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:16.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:40:16.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:40:16.572 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:16.570+0000 7f9b25e51780 -1 osd.2 42 log_to_monitors true 2026-02-25T16:40:17.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:40:17.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:17.534 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:40:17.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:40:17.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:17.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:40:17.760 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up out weight 0 up_from 48 up_thru 20 down_at 43 last_clean_interval [18,42) [v2:127.0.0.1:6818/1845582390,v1:127.0.0.1:6819/1845582390] [v2:127.0.0.1:6820/1845582390,v1:127.0.0.1:6821/1845582390] exists,up 86b39497-c715-423c-8c40-ddaed7e93a46 2026-02-25T16:40:17.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:40:17.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:40:17.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:40:17.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph osd in 2 2026-02-25T16:40:18.009 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already in. 2026-02-25T16:40:18.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: sleep 15 2026-02-25T16:40:33.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: seq 1 240 2026-02-25T16:40:33.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2026-02-25T16:40:33.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: get_state 2.0 2026-02-25T16:40:33.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2026-02-25T16:40:33.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2026-02-25T16:40:33.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2026-02-25T16:40:33.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2026-02-25T16:40:33.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2026-02-25T16:40:33.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2026-02-25T16:40:33.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:574: TEST_ec_backfill_unfound: grep backfill_unfound 2026-02-25T16:40:33.246 INFO:tasks.workunit.client.0.trial129.stdout:active+backfill_unfound+undersized+degraded+remapped 2026-02-25T16:40:33.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2026-02-25T16:40:33.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: break 2026-02-25T16:40:33.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:582: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T16:40:33.469 INFO:tasks.workunit.client.0.trial129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T16:40:33.469 INFO:tasks.workunit.client.0.trial129.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2026-02-25T16:40:18.751795+0000 46'301 54:908 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2026-02-25T16:40:03.814729+0000 0'0 2026-02-25T16:40:03.814729+0000 0 0 no scrub is scheduled 0 0 2026-02-25T16:40:33.470 INFO:tasks.workunit.client.0.trial129.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:18.047966+0000 0'0 54:61 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-27T04:22:22.547963+0000 0 0 2026-02-25T16:40:33.470 INFO:tasks.workunit.client.0.trial129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.016888+0000 0'0 54:122 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:22:19.353288+0000 0 0 2026-02-25T16:40:33.470 INFO:tasks.workunit.client.0.trial129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.419059+0000 0'0 54:43 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T20:17:11.594100+0000 0 0 2026-02-25T16:40:33.470 INFO:tasks.workunit.client.0.trial129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:18.048073+0000 0'0 54:110 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-27T00:35:41.586605+0000 0 0 2026-02-25T16:40:33.470 INFO:tasks.workunit.client.0.trial129.stdout: 2026-02-25T16:40:33.470 INFO:tasks.workunit.client.0.trial129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T16:40:33.470 INFO:tasks.workunit.client.0.trial129.stderr:dumped pgs 2026-02-25T16:40:33.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:583: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:40:33.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:40:33.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:40:33.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:40:33.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:40:33.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:40:33.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:40:33.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: sleep 5 2026-02-25T16:40:38.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:586: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-25T16:40:39.001 INFO:tasks.workunit.client.0.trial129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T16:40:39.002 INFO:tasks.workunit.client.0.trial129.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2026-02-25T16:40:33.656988+0000 46'301 56:920 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2026-02-25T16:40:03.814729+0000 0'0 2026-02-25T16:40:03.814729+0000 0 0 no scrub is scheduled 0 0 2026-02-25T16:40:39.002 INFO:tasks.workunit.client.0.trial129.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:18.047966+0000 0'0 54:61 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-27T04:22:22.547963+0000 0 0 2026-02-25T16:40:39.002 INFO:tasks.workunit.client.0.trial129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.016888+0000 0'0 54:122 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:22:19.353288+0000 0 0 2026-02-25T16:40:39.002 INFO:tasks.workunit.client.0.trial129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:40:03.419059+0000 0'0 54:43 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-26T20:17:11.594100+0000 0 0 2026-02-25T16:40:39.002 INFO:tasks.workunit.client.0.trial129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2026-02-25T16:40:33.655720+0000 0'0 56:118 [1,0] 1 [1,0] 1 0'0 2026-02-25T16:39:49.472701+0000 0'0 2026-02-25T16:39:49.472701+0000 0 0 periodic scrub scheduled @ 2026-02-27T01:52:12.883872+0000 0 0 2026-02-25T16:40:39.002 INFO:tasks.workunit.client.0.trial129.stdout: 2026-02-25T16:40:39.002 INFO:tasks.workunit.client.0.trial129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T16:40:39.003 INFO:tasks.workunit.client.0.trial129.stderr:dumped pgs 2026-02-25T16:40:39.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout:{ 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "num_missing": 1, 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "num_unfound": 1, 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "objects": [ 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "oid": { 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "oid": "obj250", 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "key": "", 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "snapid": -2, 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "hash": 2249616407, 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "max": 0, 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "pool": 2, 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "namespace": "" 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "need": "46'251", 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "have": "0'0", 2026-02-25T16:40:39.069 INFO:tasks.workunit.client.0.trial129.stdout: "flags": "none", 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "locations": [ 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: ] 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "state": "NotRecovering", 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "available_might_have_unfound": true, 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "might_have_unfound": [ 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "2(4)", 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "status": "osd is down" 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout: "more": false 2026-02-25T16:40:39.070 INFO:tasks.workunit.client.0.trial129.stdout:} 2026-02-25T16:40:39.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:588: TEST_ec_backfill_unfound: ceph pg 2.0 query 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout:{ 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: "snap_trimq": "[]", 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: "snap_trimq_len": 0, 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+recovery_unfound+undersized+degraded", 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: "epoch": 56, 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.134 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: "acting_recovery_backfill": [ 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:40:39.135 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "info": { 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s0", 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "0", 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'301", 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'250", 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "46'200", 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 301, 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:40:39.136 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "55", 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 55, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 55, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 55, 2026-02-25T16:40:39.137 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'301", 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 921, 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 56, 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+recovery_unfound+undersized+degraded", 2026-02-25T16:40:39.138 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:40:33.656988+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:40:33.656988+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:40:33.656988+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:40:33.656988+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:40:08.978386+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:40:33.655645+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:40:33.655645+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:40:33.656988+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:40:33.654322+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:40:33.654182+0000", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 55, 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "46'200", 2026-02-25T16:40:39.139 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "46'200", 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 200, 2026-02-25T16:40:39.140 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "no scrub is scheduled", 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:40:39.141 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 1232896, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 301, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 1505, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 1, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 1, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 303, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 1, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 301, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:40:39.142 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 301, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 1204, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:40:39.143 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:40:39.144 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.145 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 300 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.146 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),4(2)", 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 1 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:40:39.147 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "peer_info": [ 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "peer": "0(3)", 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s3", 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "3", 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'301", 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'301", 2026-02-25T16:40:39.148 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "46'200", 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 301, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 55, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:40:39.149 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 55, 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 55, 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'301", 2026-02-25T16:40:39.150 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 911, 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 55, 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+remapped", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:40:08.978386+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:40:09.599247+0000", 2026-02-25T16:40:39.151 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:40:09.599182+0000", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 55, 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "46'200", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "46'200", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.152 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 200, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:40:39.153 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T02:13:39.312644+0000", 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 1232896, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 301, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 1505, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 0, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 301, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:40:39.154 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 301, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 301, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 1204, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:40:39.155 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:40:39.156 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.157 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.158 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 301 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:40:39.159 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "peer": "1(1)", 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s1", 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "1", 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'301", 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'301", 2026-02-25T16:40:39.160 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "46'200", 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 301, 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 55, 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:40:39.161 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 55, 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 55, 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:40:39.162 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'301", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 332, 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 46, 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+remapped", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:40:08.978386+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:40:09.599247+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:40:09.599182+0000", 2026-02-25T16:40:39.163 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 55, 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "46'200", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "46'200", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:40:39.164 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 200, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T02:13:39.312644+0000", 2026-02-25T16:40:39.165 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 1232896, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 301, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 1505, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 1, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 301, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 0, 2026-02-25T16:40:39.166 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 301, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 301, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 1204, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:40:39.167 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:40:39.168 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.169 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.170 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 301 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:40:39.171 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "peer": "4(2)", 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s2", 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "2", 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'301", 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'301", 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "46'200", 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 301, 2026-02-25T16:40:39.172 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 55, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 55, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 55, 2026-02-25T16:40:39.173 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'301", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 332, 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 46, 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+remapped", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.174 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:40:08.978386+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:40:09.600316+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:40:15.409128+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:40:09.599247+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:40:09.599182+0000", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 55, 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "46'200", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "46'200", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:40:39.175 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:40:03.814729+0000", 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 200, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:40:39.176 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T02:13:39.312644+0000", 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 1232896, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 301, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 1505, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 301, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 0, 2026-02-25T16:40:39.177 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 301, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 301, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 1204, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:40:39.178 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.179 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: 2147483647 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:40:39.180 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 301 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 56, 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.181 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "recovery_state": [ 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "name": "Started/Primary/Active", 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "enter_time": "2026-02-25T16:40:33.654196+0000", 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "might_have_unfound": [ 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "0(3)", 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "status": "already probed" 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "1(1)", 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "status": "already probed" 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "2(4)", 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "status": "osd is down" 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.182 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "4(2)", 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "status": "already probed" 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "recovery_progress": { 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "backfill_targets": [], 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "waiting_on_backfill": [], 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill_started": "MIN", 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "backfill_info": { 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "begin": "MIN", 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "end": "MIN", 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "objects": [] 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "peer_backfill_info": [], 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "backfills_in_flight": [], 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "recovering": [], 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "pg_backend": { 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "recovery_ops": [], 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: "read_ops": [] 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.183 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "name": "Started", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "enter_time": "2026-02-25T16:40:33.554683+0000" 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "scrubber": { 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "active": false, 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "must_scrub": false, 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "must_deep_scrub": false, 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "must_repair": false, 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "sched_time": "2026-02-26T22:52:54.821883+0000", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "schedule": "not registered for scrubbing", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "urgency": "periodic-regular", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "shallow-target": { 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s0", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "level": "shallow", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "urgency": "periodic-regular", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "sched_time": "2026-02-26T22:52:54.821883+0000", 2026-02-25T16:40:39.184 INFO:tasks.workunit.client.0.trial129.stdout: "orig_sched_time": "2026-02-26T22:52:54.821883+0000", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "last_issue": "ok", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "forced": false, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "eligible": false, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "queued": false, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "active": false 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "deep-target": { 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s0", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "level": "deep", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "urgency": "periodic-regular", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "sched_time": "2026-03-04T13:22:57.406007+0000", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "orig_sched_time": "2026-03-04T13:22:57.406007+0000", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "last_issue": "ok", 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "forced": false, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "eligible": false, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "queued": false, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "active": false 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout: "agent_state": {} 2026-02-25T16:40:39.185 INFO:tasks.workunit.client.0.trial129.stdout:} 2026-02-25T16:40:39.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:40:39.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_backfill_unfound: grep -q obj250 2026-02-25T16:40:39.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:40:39.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2026-02-25T16:40:39.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:592: TEST_ec_backfill_unfound: check=true 2026-02-25T16:40:39.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: test true == true 2026-02-25T16:40:39.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:40:39.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2026-02-25T16:40:39.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2026-02-25T16:40:39.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: check='osd is down' 2026-02-25T16:40:39.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:596: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2026-02-25T16:40:39.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:40:39.333 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: check='2(4)' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:599: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:601: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:40:39.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:40:39.401 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:40:39.401 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:40:39.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:40:39.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:40:39.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:40:39.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:40:39.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:40:39.404 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:40:39.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:40:39.405 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:40:39.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:40:39.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:40:39.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:40:39.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:40:39.417 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:39.415+0000 7fb840c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:39.418 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:39.416+0000 7fb840c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:39.419 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:39.417+0000 7fb840c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:39.544 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:39.542+0000 7fb840c56780 -1 Falling back to public interface 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:39.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:40:39.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:40:39.890 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:39.889+0000 7fb840c56780 -1 osd.2 54 log_to_monitors true 2026-02-25T16:40:40.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:40:40.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:40.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:40:40.861 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:40:40.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:40.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:40:41.090 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 58 up_thru 20 down_at 55 last_clean_interval [48,54) [v2:127.0.0.1:6818/4025873325,v1:127.0.0.1:6819/4025873325] [v2:127.0.0.1:6820/4025873325,v1:127.0.0.1:6821/4025873325] exists,up 86b39497-c715-423c-8c40-ddaed7e93a46 2026-02-25T16:40:41.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:40:41.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:40:41.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:40:41.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2026-02-25T16:40:46.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:605: TEST_ec_backfill_unfound: test 124 = 124 2026-02-25T16:40:46.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:607: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2026-02-25T16:40:46.157 INFO:tasks.workunit.client.0.trial129.stderr:pg has no unfound objects 2026-02-25T16:40:46.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_backfill_unfound: wait_for_clean 2026-02-25T16:40:46.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:40:46.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:40:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:40:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:40:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:40:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:40:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:40:46.246 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:40:46.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:40:46.475 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:40:46.475 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:40:46.475 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:40:46.475 INFO:tasks.workunit.client.0.trial129.stderr:4' 2026-02-25T16:40:46.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:40:46.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:46.476 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:40:46.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803792 2026-02-25T16:40:46.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803792 2026-02-25T16:40:46.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792' 2026-02-25T16:40:46.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:46.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:40:46.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607568 2026-02-25T16:40:46.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607568 2026-02-25T16:40:46.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792 1-51539607568' 2026-02-25T16:40:46.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:46.601 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:40:46.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103172 2026-02-25T16:40:46.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103172 2026-02-25T16:40:46.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792 1-51539607568 2-249108103172' 2026-02-25T16:40:46.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:46.665 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:40:46.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117006 2026-02-25T16:40:46.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117006 2026-02-25T16:40:46.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792 1-51539607568 2-249108103172 3-115964117006' 2026-02-25T16:40:46.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:40:46.731 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:40:46.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855373 2026-02-25T16:40:46.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855373 2026-02-25T16:40:46.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803792 1-51539607568 2-249108103172 3-115964117006 4-150323855373' 2026-02-25T16:40:46.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:46.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803792 2026-02-25T16:40:46.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:46.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:40:46.796 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803792 2026-02-25T16:40:46.797 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:46.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803792 2026-02-25T16:40:46.798 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803792 2026-02-25T16:40:46.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803792' 2026-02-25T16:40:46.798 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:40:47.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803790 -lt 25769803792 2026-02-25T16:40:47.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:40:48.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:40:48.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:40:48.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803792 -lt 25769803792 2026-02-25T16:40:48.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:48.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607568 2026-02-25T16:40:48.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:48.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:40:48.250 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607568 2026-02-25T16:40:48.250 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:48.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607568 2026-02-25T16:40:48.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607568' 2026-02-25T16:40:48.252 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607568 2026-02-25T16:40:48.252 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:40:48.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607568 -lt 51539607568 2026-02-25T16:40:48.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:48.479 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-249108103172 2026-02-25T16:40:48.479 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:48.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:40:48.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-249108103172 2026-02-25T16:40:48.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:48.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=249108103172 2026-02-25T16:40:48.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 249108103172' 2026-02-25T16:40:48.482 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 249108103172 2026-02-25T16:40:48.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:40:48.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 249108103172 -lt 249108103172 2026-02-25T16:40:48.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:48.711 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117006 2026-02-25T16:40:48.712 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:48.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:40:48.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117006 2026-02-25T16:40:48.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:48.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117006 2026-02-25T16:40:48.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117006' 2026-02-25T16:40:48.714 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964117006 2026-02-25T16:40:48.715 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:40:48.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117006 -lt 115964117006 2026-02-25T16:40:48.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:40:48.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855373 2026-02-25T16:40:48.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:40:48.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:40:48.940 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855373 2026-02-25T16:40:48.940 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:40:48.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855373 2026-02-25T16:40:48.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855373' 2026-02-25T16:40:48.942 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855373 2026-02-25T16:40:48.942 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:40:49.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855373 -lt 150323855373 2026-02-25T16:40:49.166 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:40:49.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:40:49.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:40:49.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:40:49.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:40:49.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:40:49.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:40:49.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:40:49.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:40:49.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:40:49.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:40:49.705 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:40:49.705 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:40:49.705 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:40:49.705 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:40:50.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:40:50.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:40:50.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:40:50.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: seq 1 300 2026-02-25T16:40:50.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2026-02-25T16:40:50.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2026-02-25T16:40:50.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.054 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.054 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2026-02-25T16:40:50.054 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2026-02-25T16:40:50.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2026-02-25T16:40:50.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.104 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.104 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2026-02-25T16:40:50.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2026-02-25T16:40:50.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2026-02-25T16:40:50.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2026-02-25T16:40:50.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2026-02-25T16:40:50.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2026-02-25T16:40:50.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2026-02-25T16:40:50.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2026-02-25T16:40:50.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2026-02-25T16:40:50.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2026-02-25T16:40:50.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2026-02-25T16:40:50.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2026-02-25T16:40:50.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2026-02-25T16:40:50.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2026-02-25T16:40:50.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2026-02-25T16:40:50.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2026-02-25T16:40:50.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2026-02-25T16:40:50.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2026-02-25T16:40:50.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2026-02-25T16:40:50.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2026-02-25T16:40:50.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2026-02-25T16:40:50.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2026-02-25T16:40:50.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2026-02-25T16:40:50.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2026-02-25T16:40:50.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2026-02-25T16:40:50.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2026-02-25T16:40:50.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2026-02-25T16:40:50.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2026-02-25T16:40:50.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.579 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.580 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.580 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2026-02-25T16:40:50.580 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.596 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2026-02-25T16:40:50.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.614 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.614 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2026-02-25T16:40:50.614 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2026-02-25T16:40:50.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2026-02-25T16:40:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2026-02-25T16:40:50.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2026-02-25T16:40:50.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2026-02-25T16:40:50.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2026-02-25T16:40:50.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2026-02-25T16:40:50.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2026-02-25T16:40:50.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2026-02-25T16:40:50.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2026-02-25T16:40:50.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2026-02-25T16:40:50.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2026-02-25T16:40:50.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2026-02-25T16:40:50.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2026-02-25T16:40:50.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2026-02-25T16:40:50.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2026-02-25T16:40:50.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2026-02-25T16:40:50.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2026-02-25T16:40:50.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2026-02-25T16:40:50.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2026-02-25T16:40:50.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2026-02-25T16:40:50.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:50.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:50.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2026-02-25T16:40:50.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2026-02-25T16:40:50.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2026-02-25T16:40:51.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2026-02-25T16:40:51.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2026-02-25T16:40:51.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2026-02-25T16:40:51.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2026-02-25T16:40:51.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2026-02-25T16:40:51.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2026-02-25T16:40:51.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2026-02-25T16:40:51.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2026-02-25T16:40:51.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2026-02-25T16:40:51.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2026-02-25T16:40:51.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.182 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2026-02-25T16:40:51.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.198 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2026-02-25T16:40:51.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2026-02-25T16:40:51.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2026-02-25T16:40:51.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2026-02-25T16:40:51.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2026-02-25T16:40:51.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.283 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2026-02-25T16:40:51.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2026-02-25T16:40:51.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2026-02-25T16:40:51.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2026-02-25T16:40:51.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2026-02-25T16:40:51.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2026-02-25T16:40:51.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.383 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2026-02-25T16:40:51.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2026-02-25T16:40:51.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2026-02-25T16:40:51.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2026-02-25T16:40:51.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.450 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.451 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.451 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2026-02-25T16:40:51.451 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2026-02-25T16:40:51.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2026-02-25T16:40:51.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2026-02-25T16:40:51.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.517 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2026-02-25T16:40:51.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2026-02-25T16:40:51.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2026-02-25T16:40:51.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2026-02-25T16:40:51.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2026-02-25T16:40:51.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2026-02-25T16:40:51.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2026-02-25T16:40:51.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2026-02-25T16:40:51.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2026-02-25T16:40:51.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2026-02-25T16:40:51.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2026-02-25T16:40:51.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2026-02-25T16:40:51.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2026-02-25T16:40:51.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2026-02-25T16:40:51.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2026-02-25T16:40:51.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2026-02-25T16:40:51.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2026-02-25T16:40:51.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2026-02-25T16:40:51.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2026-02-25T16:40:51.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2026-02-25T16:40:51.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2026-02-25T16:40:51.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.864 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2026-02-25T16:40:51.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2026-02-25T16:40:51.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2026-02-25T16:40:51.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2026-02-25T16:40:51.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2026-02-25T16:40:51.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2026-02-25T16:40:51.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2026-02-25T16:40:51.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2026-02-25T16:40:51.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2026-02-25T16:40:51.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:51.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:51.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2026-02-25T16:40:51.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2026-02-25T16:40:52.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2026-02-25T16:40:52.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2026-02-25T16:40:52.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2026-02-25T16:40:52.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2026-02-25T16:40:52.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2026-02-25T16:40:52.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2026-02-25T16:40:52.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2026-02-25T16:40:52.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2026-02-25T16:40:52.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2026-02-25T16:40:52.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2026-02-25T16:40:52.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2026-02-25T16:40:52.195 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2026-02-25T16:40:52.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2026-02-25T16:40:52.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2026-02-25T16:40:52.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2026-02-25T16:40:52.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2026-02-25T16:40:52.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2026-02-25T16:40:52.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2026-02-25T16:40:52.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2026-02-25T16:40:52.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2026-02-25T16:40:52.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2026-02-25T16:40:52.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2026-02-25T16:40:52.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2026-02-25T16:40:52.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2026-02-25T16:40:52.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.427 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2026-02-25T16:40:52.427 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2026-02-25T16:40:52.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2026-02-25T16:40:52.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2026-02-25T16:40:52.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2026-02-25T16:40:52.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2026-02-25T16:40:52.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2026-02-25T16:40:52.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2026-02-25T16:40:52.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2026-02-25T16:40:52.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.578 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.578 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2026-02-25T16:40:52.578 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2026-02-25T16:40:52.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2026-02-25T16:40:52.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2026-02-25T16:40:52.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2026-02-25T16:40:52.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2026-02-25T16:40:52.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2026-02-25T16:40:52.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2026-02-25T16:40:52.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2026-02-25T16:40:52.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2026-02-25T16:40:52.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2026-02-25T16:40:52.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2026-02-25T16:40:52.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2026-02-25T16:40:52.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2026-02-25T16:40:52.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2026-02-25T16:40:52.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2026-02-25T16:40:52.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2026-02-25T16:40:52.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2026-02-25T16:40:52.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.877 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2026-02-25T16:40:52.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2026-02-25T16:40:52.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2026-02-25T16:40:52.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2026-02-25T16:40:52.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2026-02-25T16:40:52.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.960 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2026-02-25T16:40:52.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2026-02-25T16:40:52.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2026-02-25T16:40:52.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:52.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:52.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2026-02-25T16:40:52.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2026-02-25T16:40:53.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2026-02-25T16:40:53.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2026-02-25T16:40:53.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2026-02-25T16:40:53.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2026-02-25T16:40:53.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2026-02-25T16:40:53.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2026-02-25T16:40:53.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2026-02-25T16:40:53.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2026-02-25T16:40:53.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2026-02-25T16:40:53.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2026-02-25T16:40:53.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2026-02-25T16:40:53.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2026-02-25T16:40:53.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2026-02-25T16:40:53.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2026-02-25T16:40:53.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2026-02-25T16:40:53.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.274 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2026-02-25T16:40:53.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2026-02-25T16:40:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2026-02-25T16:40:53.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2026-02-25T16:40:53.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2026-02-25T16:40:53.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2026-02-25T16:40:53.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2026-02-25T16:40:53.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2026-02-25T16:40:53.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2026-02-25T16:40:53.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2026-02-25T16:40:53.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2026-02-25T16:40:53.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2026-02-25T16:40:53.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2026-02-25T16:40:53.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2026-02-25T16:40:53.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2026-02-25T16:40:53.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2026-02-25T16:40:53.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2026-02-25T16:40:53.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2026-02-25T16:40:53.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.575 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2026-02-25T16:40:53.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2026-02-25T16:40:53.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2026-02-25T16:40:53.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2026-02-25T16:40:53.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.641 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2026-02-25T16:40:53.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2026-02-25T16:40:53.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2026-02-25T16:40:53.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2026-02-25T16:40:53.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2026-02-25T16:40:53.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2026-02-25T16:40:53.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2026-02-25T16:40:53.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2026-02-25T16:40:53.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2026-02-25T16:40:53.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2026-02-25T16:40:53.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.808 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.808 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2026-02-25T16:40:53.808 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2026-02-25T16:40:53.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2026-02-25T16:40:53.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2026-02-25T16:40:53.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2026-02-25T16:40:53.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2026-02-25T16:40:53.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2026-02-25T16:40:53.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2026-02-25T16:40:53.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2026-02-25T16:40:53.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2026-02-25T16:40:53.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2026-02-25T16:40:53.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2026-02-25T16:40:53.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:53.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:53.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2026-02-25T16:40:53.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2026-02-25T16:40:54.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2026-02-25T16:40:54.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2026-02-25T16:40:54.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.055 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2026-02-25T16:40:54.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2026-02-25T16:40:54.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2026-02-25T16:40:54.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2026-02-25T16:40:54.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2026-02-25T16:40:54.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2026-02-25T16:40:54.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2026-02-25T16:40:54.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2026-02-25T16:40:54.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.187 INFO:tasks.workunit.client.0.trial129.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2026-02-25T16:40:54.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2026-02-25T16:40:54.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2026-02-25T16:40:54.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2026-02-25T16:40:54.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2026-02-25T16:40:54.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2026-02-25T16:40:54.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2026-02-25T16:40:54.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2026-02-25T16:40:54.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2026-02-25T16:40:54.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2026-02-25T16:40:54.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2026-02-25T16:40:54.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.354 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2026-02-25T16:40:54.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2026-02-25T16:40:54.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2026-02-25T16:40:54.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2026-02-25T16:40:54.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2026-02-25T16:40:54.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2026-02-25T16:40:54.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2026-02-25T16:40:54.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2026-02-25T16:40:54.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2026-02-25T16:40:54.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2026-02-25T16:40:54.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2026-02-25T16:40:54.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2026-02-25T16:40:54.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2026-02-25T16:40:54.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.572 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.572 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2026-02-25T16:40:54.572 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2026-02-25T16:40:54.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2026-02-25T16:40:54.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2026-02-25T16:40:54.623 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2026-02-25T16:40:54.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2026-02-25T16:40:54.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2026-02-25T16:40:54.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2026-02-25T16:40:54.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.705 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2026-02-25T16:40:54.706 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2026-02-25T16:40:54.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2026-02-25T16:40:54.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2026-02-25T16:40:54.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2026-02-25T16:40:54.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2026-02-25T16:40:54.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2026-02-25T16:40:54.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2026-02-25T16:40:54.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2026-02-25T16:40:54.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2026-02-25T16:40:54.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2026-02-25T16:40:54.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2026-02-25T16:40:54.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2026-02-25T16:40:54.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2026-02-25T16:40:54.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2026-02-25T16:40:54.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2026-02-25T16:40:54.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2026-02-25T16:40:54.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2026-02-25T16:40:54.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:54.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:54.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2026-02-25T16:40:54.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2026-02-25T16:40:55.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:55.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:40:55.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2026-02-25T16:40:55.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2026-02-25T16:40:55.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:55.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:40:55.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:40:55.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:40:55.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:40:55.300 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:40:55.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:40:55.609 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:40:55.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:40:55.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:40:55.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:40:55.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:40:55.617 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:40:55.617 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:40:55.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:40:55.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:40:55.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:40:55.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:40:55.725 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:40:55.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:40:55.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:40:55.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:40:55.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:40:55.728 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:40:55.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:40:55.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:40:55.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:40:55.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:40:55.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:40:55.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:40:55.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:40:55.766 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:40:55.766 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:55.766 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:55.766 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:40:55.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:40:55.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:40:55.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:40:55.769 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:40:55.769 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:40:55.770 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:40:55.770 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:40:55.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:40:55.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:40:55.772 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:40:55.772 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:40:55.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:40:55.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:40:55.773 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:40:55.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:40:55.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:40:55.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:40:55.776 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:40:55.776 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:55.776 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:55.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:40:55.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:40:55.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:40:55.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:40:55.778 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:40:55.778 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:55.778 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:55.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:40:55.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:40:55.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:40:55.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:40:55.798 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:40:55.798 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:40:55.798 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:40:55.799 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:40:55.799 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:55.799 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:55.799 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:55.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:40:55.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:40:55.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:40:55.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:40:55.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:40:55.823 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:40:55.824 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:40:55.824 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:40:55.824 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:40:55.824 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:40:55.825 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:55.825 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:55.825 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:55.825 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:40:55.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:40:55.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:40:55.867 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:40:55.868 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:55.868 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:55.868 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:55.868 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:40:55.868 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:40:55.868 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:40:55.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:40:55.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:40:55.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:40:55.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:40:55.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:40:55.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:40:55.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:40:56.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:40:56.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:40:56.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:40:56.010 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:40:56.010 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:56.010 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:56.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:56.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:40:56.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:40:56.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:40:56.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:40:56.118 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:40:56.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:40:57.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:40:57.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:40:57.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:40:57.128 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:40:57.128 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:57.128 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:57.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:40:57.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:40:57.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:40:57.166 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:40:57.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:40:57.172 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:40:55.814+0000 7f229261bc00 0 load: jerasure load: lrc 2026-02-25T16:40:57.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2026-02-25T16:40:57.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:379: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2026-02-25T16:40:57.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:380: TEST_ec_object_attr_read_error: local objname=myobject 2026-02-25T16:40:57.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:382: TEST_ec_object_attr_read_error: setup_osds 7 2026-02-25T16:40:57.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T16:40:57.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:40:57.173 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T16:40:57.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:40:57.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:40:57.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:40:57.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:40:57.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:40:57.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:40:57.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:40:57.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:40:57.178 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:57.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:40:57.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e0da1ba0-8121-4fa8-a4c0-761662006e6f 2026-02-25T16:40:57.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e0da1ba0-8121-4fa8-a4c0-761662006e6f' 2026-02-25T16:40:57.181 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 e0da1ba0-8121-4fa8-a4c0-761662006e6f 2026-02-25T16:40:57.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:40:57.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZJp9pm6siCxAAqkrJKAKj58n6atIWCvhmzQ== 2026-02-25T16:40:57.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZJp9pm6siCxAAqkrJKAKj58n6atIWCvhmzQ=="}' 2026-02-25T16:40:57.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e0da1ba0-8121-4fa8-a4c0-761662006e6f -i td/test-erasure-eio/0/new.json 2026-02-25T16:40:57.291 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:40:57.298 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:40:57.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZJp9pm6siCxAAqkrJKAKj58n6atIWCvhmzQ== --osd-uuid e0da1ba0-8121-4fa8-a4c0-761662006e6f 2026-02-25T16:40:57.315 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.313+0000 7f6b7fffd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:57.316 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.314+0000 7f6b7fffd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:57.317 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.315+0000 7f6b7fffd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:57.317 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.316+0000 7f6b7fffd780 -1 bdev(0x3da4c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:40:57.318 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.316+0000 7f6b7fffd780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:40:57.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:40:57.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:40:57.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:40:57.725 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:40:57.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:40:57.826 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:40:57.826 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:40:57.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:40:57.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:40:57.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:40:57.827 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:40:57.840 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.839+0000 7f67fec41780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:57.842 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.840+0000 7f67fec41780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:57.843 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:57.841+0000 7f67fec41780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:40:57.921 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:40:57.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:57.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:40:58.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:40:58.049 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:58.048+0000 7f67fec41780 -1 Falling back to public interface 2026-02-25T16:40:58.407 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:58.405+0000 7f67fec41780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:40:59.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:40:59.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:40:59.019 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:40:59.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:40:59.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:40:59.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:40:59.248 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/541279435,v1:127.0.0.1:6803/541279435] [v2:127.0.0.1:6804/541279435,v1:127.0.0.1:6805/541279435] exists,up e0da1ba0-8121-4fa8-a4c0-761662006e6f 2026-02-25T16:40:59.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:40:59.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:40:59.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:40:59.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:40:59.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:40:59.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:40:59.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=921a5978-1407-43a0-b445-00833e7b7202 2026-02-25T16:40:59.256 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 921a5978-1407-43a0-b445-00833e7b7202 2026-02-25T16:40:59.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 921a5978-1407-43a0-b445-00833e7b7202' 2026-02-25T16:40:59.256 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:40:59.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbJp9pIqiKDxAASFzrWHp5B4nd5YYckJEgUQ== 2026-02-25T16:40:59.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbJp9pIqiKDxAASFzrWHp5B4nd5YYckJEgUQ=="}' 2026-02-25T16:40:59.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 921a5978-1407-43a0-b445-00833e7b7202 -i td/test-erasure-eio/1/new.json 2026-02-25T16:40:59.492 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:40:59.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:40:59.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbJp9pIqiKDxAASFzrWHp5B4nd5YYckJEgUQ== --osd-uuid 921a5978-1407-43a0-b445-00833e7b7202 2026-02-25T16:40:59.515 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:59.513+0000 7f199884b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:59.516 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:59.515+0000 7f199884b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:59.517 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:59.516+0000 7f199884b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:40:59.517 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:59.516+0000 7f199884b780 -1 bdev(0x2a16c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:40:59.517 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:40:59.516+0000 7f199884b780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:40:59.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:40:59.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:40:59.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:40:59.876 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:40:59.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:00.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:41:00.208 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:41:00.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:00.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:00.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:00.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:00.222 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:00.221+0000 7f0188209780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:00.223 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:00.222+0000 7f0188209780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:00.225 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:00.223+0000 7f0188209780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:00.357 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:00.355+0000 7f0188209780 -1 Falling back to public interface 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:00.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:00.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:00.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:41:00.641 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:00.639+0000 7f0188209780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:41:00.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:01.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:01.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:01.658 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:01.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:01.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:01.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:41:01.882 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/209550332,v1:127.0.0.1:6811/209550332] [v2:127.0.0.1:6812/209550332,v1:127.0.0.1:6813/209550332] exists,up 921a5978-1407-43a0-b445-00833e7b7202 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:41:01.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:01.888 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:01.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=48ef2187-726b-4563-90ca-1547c160707b 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 48ef2187-726b-4563-90ca-1547c160707b' 2026-02-25T16:41:01.890 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 48ef2187-726b-4563-90ca-1547c160707b 2026-02-25T16:41:01.891 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:01.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdJp9pmb1jNRAAYAQta/epdxPAeJb0iHfX1Q== 2026-02-25T16:41:01.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdJp9pmb1jNRAAYAQta/epdxPAeJb0iHfX1Q=="}' 2026-02-25T16:41:01.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 48ef2187-726b-4563-90ca-1547c160707b -i td/test-erasure-eio/2/new.json 2026-02-25T16:41:02.124 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:41:02.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:41:02.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdJp9pmb1jNRAAYAQta/epdxPAeJb0iHfX1Q== --osd-uuid 48ef2187-726b-4563-90ca-1547c160707b 2026-02-25T16:41:02.148 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.147+0000 7f672d658780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:02.150 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.148+0000 7f672d658780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:02.151 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.149+0000 7f672d658780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:02.151 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.149+0000 7f672d658780 -1 bdev(0x279ec00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:02.151 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.149+0000 7f672d658780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:41:02.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:41:02.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:02.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:41:02.493 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:41:02.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:02.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:41:02.807 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:41:02.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:02.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:02.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:02.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:02.821 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.820+0000 7f27a8a2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:02.823 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.821+0000 7f27a8a2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:02.823 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.822+0000 7f27a8a2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:02.962 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:02.960+0000 7f27a8a2c780 -1 Falling back to public interface 2026-02-25T16:41:03.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:41:03.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:03.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:41:03.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:03.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:03.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:03.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:03.032 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:03.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:03.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:41:03.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:03.342 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:03.341+0000 7f27a8a2c780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:41:04.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:04.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:04.256 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:04.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:04.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:04.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:41:04.486 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1396210365,v1:127.0.0.1:6819/1396210365] [v2:127.0.0.1:6820/1396210365,v1:127.0.0.1:6821/1396210365] exists,up 48ef2187-726b-4563-90ca-1547c160707b 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:04.488 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:41:04.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2abc883d-9507-4d62-866e-994d63a5b3cc 2026-02-25T16:41:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 2abc883d-9507-4d62-866e-994d63a5b3cc' 2026-02-25T16:41:04.490 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 2abc883d-9507-4d62-866e-994d63a5b3cc 2026-02-25T16:41:04.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:04.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAgJp9p19vIHRAAn7TdyIJe4LTOYpeMeMhnOg== 2026-02-25T16:41:04.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAgJp9p19vIHRAAn7TdyIJe4LTOYpeMeMhnOg=="}' 2026-02-25T16:41:04.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2abc883d-9507-4d62-866e-994d63a5b3cc -i td/test-erasure-eio/3/new.json 2026-02-25T16:41:04.729 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:41:04.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:41:04.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgJp9p19vIHRAAn7TdyIJe4LTOYpeMeMhnOg== --osd-uuid 2abc883d-9507-4d62-866e-994d63a5b3cc 2026-02-25T16:41:04.751 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:04.750+0000 7f089942c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:04.752 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:04.751+0000 7f089942c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:04.753 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:04.752+0000 7f089942c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:04.754 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:04.752+0000 7f089942c780 -1 bdev(0x3f5ac00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:04.754 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:04.752+0000 7f089942c780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:41:05.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:41:05.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:05.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:41:05.113 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:41:05.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:05.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:41:05.432 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:41:05.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:05.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:05.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:05.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:05.446 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:05.444+0000 7f9a4b22c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:05.447 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:05.445+0000 7f9a4b22c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:05.448 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:05.446+0000 7f9a4b22c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:05.568 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:05.566+0000 7f9a4b22c780 -1 Falling back to public interface 2026-02-25T16:41:05.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:05.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:41:05.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:05.889 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:05.888+0000 7f9a4b22c780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:41:06.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:06.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:06.884 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:06.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:06.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:06.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:41:07.109 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/471759928,v1:127.0.0.1:6827/471759928] [v2:127.0.0.1:6828/471759928,v1:127.0.0.1:6829/471759928] exists,up 2abc883d-9507-4d62-866e-994d63a5b3cc 2026-02-25T16:41:07.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:07.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:07.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:07.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T16:41:07.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:07.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:07.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=141bdccd-30b9-42db-abe9-c3be7ff5d19d 2026-02-25T16:41:07.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 141bdccd-30b9-42db-abe9-c3be7ff5d19d' 2026-02-25T16:41:07.117 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 141bdccd-30b9-42db-abe9-c3be7ff5d19d 2026-02-25T16:41:07.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:07.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAjJp9ponFCBxAAt819ywy9JDeGLUXExxNF9g== 2026-02-25T16:41:07.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAjJp9ponFCBxAAt819ywy9JDeGLUXExxNF9g=="}' 2026-02-25T16:41:07.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 141bdccd-30b9-42db-abe9-c3be7ff5d19d -i td/test-erasure-eio/4/new.json 2026-02-25T16:41:07.353 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:41:07.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T16:41:07.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjJp9ponFCBxAAt819ywy9JDeGLUXExxNF9g== --osd-uuid 141bdccd-30b9-42db-abe9-c3be7ff5d19d 2026-02-25T16:41:07.377 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:07.375+0000 7ff4a7632780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:07.378 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:07.377+0000 7ff4a7632780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:07.379 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:07.378+0000 7ff4a7632780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:07.380 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:07.378+0000 7ff4a7632780 -1 bdev(0x3490c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:07.380 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:07.378+0000 7ff4a7632780 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T16:41:07.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T16:41:07.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:07.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:41:07.709 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:41:07.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:08.028 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:41:08.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:41:08.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:08.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:08.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:08.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:08.042 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:08.041+0000 7fa884244780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:08.043 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:08.042+0000 7fa884244780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:08.044 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:08.043+0000 7fa884244780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:08.193 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:08.192+0000 7fa884244780 -1 Falling back to public interface 2026-02-25T16:41:08.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:41:08.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:08.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:41:08.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:08.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:08.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:08.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:08.260 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:08.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:08.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:41:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:08.523 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:08.521+0000 7fa884244780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:41:09.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:09.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:09.485 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:09.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:09.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:09.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:41:09.709 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 35 up_thru 36 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2479791128,v1:127.0.0.1:6835/2479791128] [v2:127.0.0.1:6836/2479791128,v1:127.0.0.1:6837/2479791128] exists,up 141bdccd-30b9-42db-abe9-c3be7ff5d19d 2026-02-25T16:41:09.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:09.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:09.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:09.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:09.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T16:41:09.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:09.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:09.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:09.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:09.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T16:41:09.715 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:09.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=39a15a3a-1aa5-474f-b9b3-aa349bcafc0c 2026-02-25T16:41:09.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 39a15a3a-1aa5-474f-b9b3-aa349bcafc0c' 2026-02-25T16:41:09.715 INFO:tasks.workunit.client.0.trial129.stdout:add osd5 39a15a3a-1aa5-474f-b9b3-aa349bcafc0c 2026-02-25T16:41:09.716 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:09.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlJp9psusEKxAASTqliCe+BSNPiDVd4j4ZyA== 2026-02-25T16:41:09.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlJp9psusEKxAASTqliCe+BSNPiDVd4j4ZyA=="}' 2026-02-25T16:41:09.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 39a15a3a-1aa5-474f-b9b3-aa349bcafc0c -i td/test-erasure-eio/5/new.json 2026-02-25T16:41:09.954 INFO:tasks.workunit.client.0.trial129.stdout:5 2026-02-25T16:41:09.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T16:41:09.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlJp9psusEKxAASTqliCe+BSNPiDVd4j4ZyA== --osd-uuid 39a15a3a-1aa5-474f-b9b3-aa349bcafc0c 2026-02-25T16:41:09.978 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:09.976+0000 7f573be2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:09.979 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:09.977+0000 7f573be2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:09.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:09.978+0000 7f573be2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:09.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:09.979+0000 7f573be2f780 -1 bdev(0x301ec00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:09.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:09.979+0000 7f573be2f780 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T16:41:10.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T16:41:10.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:10.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T16:41:10.367 INFO:tasks.workunit.client.0.trial129.stdout:adding osd5 key to auth repository 2026-02-25T16:41:10.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:10.686 INFO:tasks.workunit.client.0.trial129.stdout:start osd.5 2026-02-25T16:41:10.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T16:41:10.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:10.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:10.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:10.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:10.700 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:10.698+0000 7f780fc46780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:10.701 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:10.700+0000 7f780fc46780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:10.702 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:10.701+0000 7f780fc46780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:10.868 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:10.866+0000 7f780fc46780 -1 Falling back to public interface 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:10.910 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:10.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:10.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:41:11.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:11.222 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:11.220+0000 7f780fc46780 -1 osd.5 0 log_to_monitors true 2026-02-25T16:41:12.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:12.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:12.138 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:12.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:12.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:12.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:41:12.363 INFO:tasks.workunit.client.0.trial129.stdout:osd.5 up in weight 1 up_from 44 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/546797827,v1:127.0.0.1:6843/546797827] [v2:127.0.0.1:6844/546797827,v1:127.0.0.1:6845/546797827] exists,up 39a15a3a-1aa5-474f-b9b3-aa349bcafc0c 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:12.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:12.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:12.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:12.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:12.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T16:41:12.366 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:12.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ebeb898d-a2d8-4a4d-ad8c-cbb3f8de9707 2026-02-25T16:41:12.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 ebeb898d-a2d8-4a4d-ad8c-cbb3f8de9707' 2026-02-25T16:41:12.367 INFO:tasks.workunit.client.0.trial129.stdout:add osd6 ebeb898d-a2d8-4a4d-ad8c-cbb3f8de9707 2026-02-25T16:41:12.367 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:12.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoJp9p5+dtFhAAUUORlCzVxdMSeu/GiOdCBQ== 2026-02-25T16:41:12.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoJp9p5+dtFhAAUUORlCzVxdMSeu/GiOdCBQ=="}' 2026-02-25T16:41:12.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ebeb898d-a2d8-4a4d-ad8c-cbb3f8de9707 -i td/test-erasure-eio/6/new.json 2026-02-25T16:41:12.604 INFO:tasks.workunit.client.0.trial129.stdout:6 2026-02-25T16:41:12.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T16:41:12.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoJp9p5+dtFhAAUUORlCzVxdMSeu/GiOdCBQ== --osd-uuid ebeb898d-a2d8-4a4d-ad8c-cbb3f8de9707 2026-02-25T16:41:12.627 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:12.626+0000 7f1af07e1780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:12.629 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:12.627+0000 7f1af07e1780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:12.629 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:12.628+0000 7f1af07e1780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:12.630 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:12.628+0000 7f1af07e1780 -1 bdev(0x2c1ec00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:12.630 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:12.629+0000 7f1af07e1780 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T16:41:12.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T16:41:12.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:12.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T16:41:12.997 INFO:tasks.workunit.client.0.trial129.stdout:adding osd6 key to auth repository 2026-02-25T16:41:12.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:13.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T16:41:13.317 INFO:tasks.workunit.client.0.trial129.stdout:start osd.6 2026-02-25T16:41:13.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:13.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:13.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:13.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:13.331 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:13.329+0000 7fd9d1410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:13.332 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:13.331+0000 7fd9d1410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:13.333 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:13.332+0000 7fd9d1410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:13.527 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:13.525+0000 7fd9d1410780 -1 Falling back to public interface 2026-02-25T16:41:13.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T16:41:13.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:13.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T16:41:13.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:13.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:13.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:13.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:13.549 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:13.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:13.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:41:13.772 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:13.921 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:13.920+0000 7fd9d1410780 -1 osd.6 0 log_to_monitors true 2026-02-25T16:41:14.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:14.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:14.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:14.775 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:14.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:14.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:41:14.998 INFO:tasks.workunit.client.0.trial129.stdout:osd.6 up in weight 1 up_from 53 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1759194615,v1:127.0.0.1:6851/1759194615] [v2:127.0.0.1:6852/1759194615,v1:127.0.0.1:6853/1759194615] exists,up ebeb898d-a2d8-4a4d-ad8c-cbb3f8de9707 2026-02-25T16:41:14.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:14.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:41:15.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:41:15.042 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:41:15.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:40:58.050+0000 7f67fec41780 0 load: jerasure load: lrc 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:384: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:41:15.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T16:41:15.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:41:15.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:41:15.704 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:41:15.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:41:16.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:41:16.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:41:16.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:41:16.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:41:16.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:41:16.714 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:41:16.714 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:41:16.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:41:16.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:41:16.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:41:16.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:41:16.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:41:16.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:41:16.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:41:16.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:41:16.801 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:41:17.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:41:17.027 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:41:17.027 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:41:17.027 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:41:17.027 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:41:17.027 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:41:17.027 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:41:17.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:41:17.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:17.028 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:41:17.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803783 2026-02-25T16:41:17.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803783 2026-02-25T16:41:17.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783' 2026-02-25T16:41:17.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:17.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:41:17.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-25T16:41:17.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-25T16:41:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574855' 2026-02-25T16:41:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:17.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:41:17.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-25T16:41:17.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-25T16:41:17.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574855 2-77309411334' 2026-02-25T16:41:17.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:17.220 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:41:17.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-25T16:41:17.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-25T16:41:17.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574855 2-77309411334 3-115964116997' 2026-02-25T16:41:17.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:17.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:41:17.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855364 2026-02-25T16:41:17.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855364 2026-02-25T16:41:17.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574855 2-77309411334 3-115964116997 4-150323855364' 2026-02-25T16:41:17.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:17.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:41:17.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561029 2026-02-25T16:41:17.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561029 2026-02-25T16:41:17.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574855 2-77309411334 3-115964116997 4-150323855364 5-188978561029' 2026-02-25T16:41:17.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:17.413 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:41:17.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266691 2026-02-25T16:41:17.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266691 2026-02-25T16:41:17.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-55834574855 2-77309411334 3-115964116997 4-150323855364 5-188978561029 6-227633266691' 2026-02-25T16:41:17.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:17.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803783 2026-02-25T16:41:17.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:17.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:41:17.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803783 2026-02-25T16:41:17.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:17.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803783 2026-02-25T16:41:17.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803783' 2026-02-25T16:41:17.481 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803783 2026-02-25T16:41:17.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:41:17.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803783 2026-02-25T16:41:17.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:41:18.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:41:18.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:41:18.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803783 2026-02-25T16:41:18.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:18.937 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574855 2026-02-25T16:41:18.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:18.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:41:18.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574855 2026-02-25T16:41:18.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:18.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574855 2026-02-25T16:41:18.940 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574855 2026-02-25T16:41:18.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574855' 2026-02-25T16:41:18.941 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:41:19.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-25T16:41:19.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:19.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411334 2026-02-25T16:41:19.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:19.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:41:19.165 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411334 2026-02-25T16:41:19.165 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:19.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411334 2026-02-25T16:41:19.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411334' 2026-02-25T16:41:19.166 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411334 2026-02-25T16:41:19.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:41:19.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411334 -lt 77309411334 2026-02-25T16:41:19.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:19.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116997 2026-02-25T16:41:19.393 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:19.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:41:19.394 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116997 2026-02-25T16:41:19.394 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:19.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116997 2026-02-25T16:41:19.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116997' 2026-02-25T16:41:19.396 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116997 2026-02-25T16:41:19.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:41:19.620 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-25T16:41:19.620 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:19.621 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855364 2026-02-25T16:41:19.621 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:19.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:41:19.622 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855364 2026-02-25T16:41:19.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:19.623 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855364 2026-02-25T16:41:19.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855364' 2026-02-25T16:41:19.624 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855364 2026-02-25T16:41:19.624 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:41:19.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855364 -lt 150323855364 2026-02-25T16:41:19.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:19.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-188978561029 2026-02-25T16:41:19.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:19.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:41:19.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-188978561029 2026-02-25T16:41:19.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:19.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561029 2026-02-25T16:41:19.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 188978561029' 2026-02-25T16:41:19.850 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 188978561029 2026-02-25T16:41:19.850 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:41:20.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561029 -lt 188978561029 2026-02-25T16:41:20.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:20.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-227633266691 2026-02-25T16:41:20.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:20.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:41:20.076 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-227633266691 2026-02-25T16:41:20.076 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:20.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=227633266691 2026-02-25T16:41:20.077 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 227633266691 2026-02-25T16:41:20.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 227633266691' 2026-02-25T16:41:20.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:41:20.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 227633266691 -lt 227633266691 2026-02-25T16:41:20.306 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:41:20.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:41:20.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:41:20.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:41:20.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:41:20.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:41:20.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:41:20.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:41:20.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:41:20.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:41:20.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:41:20.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:41:20.844 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:41:20.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:41:20.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:41:21.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:41:21.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:41:21.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:41:21.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2026-02-25T16:41:21.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=pool-jerasure 2026-02-25T16:41:21.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=myobject 2026-02-25T16:41:21.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:41:21.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T16:41:21.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:387: TEST_ec_object_attr_read_error: local primary_osd=3 2026-02-25T16:41:21.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:389: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T16:41:21.384 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:41:21.384 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:41:21.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:41:21.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:41:21.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:392: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:41:21.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:41:21.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:41:21.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:41:21.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:41:21.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:41:21.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:41:21.692 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T16:41:22.383 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=mdata 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:41:22.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T16:41:22.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:41:22.385 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T16:41:22.385 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:41:22.385 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:41:22.385 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:41:22.385 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:41:22.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2147483647 2026-02-25T16:41:22.612 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:41:22.612 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:41:22.612 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:41:22.612 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:41:22.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2147483647 5 1 0 6 2026-02-25T16:41:22.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=5 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/5/type 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=5 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:41:22.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:41:22.614 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:41:22.614 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.5 2026-02-25T16:41:22.614 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2026-02-25T16:41:22.614 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2026-02-25T16:41:22.614 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:41:22.615 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:22.615 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:22.615 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.5.asok 2026-02-25T16:41:22.615 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:41:22.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:41:22.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:41:22.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:41:22.666 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.5 2026-02-25T16:41:22.666 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.5 2026-02-25T16:41:22.666 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.5 ']' 2026-02-25T16:41:22.667 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:41:22.667 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:22.667 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:22.667 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.5.asok 2026-02-25T16:41:22.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:41:22.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2026-02-25T16:41:22.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2026-02-25T16:41:22.715 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:22.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:22.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T16:41:22.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T16:41:22.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:41:22.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:41:22.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:41:22.732 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:22.730+0000 7f4c2ca62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:22.733 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:22.732+0000 7f4c2ca62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:22.734 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:22.733+0000 7f4c2ca62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:22.856 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:22.854+0000 7f4c2ca62780 -1 Falling back to public interface 2026-02-25T16:41:22.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T16:41:22.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:22.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:41:22.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:22.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:22.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:22.947 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:22.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:22.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:22.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:41:23.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:23.222 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:23.221+0000 7f4c2ca62780 -1 osd.3 60 log_to_monitors true 2026-02-25T16:41:24.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:24.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:24.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:24.173 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:24.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:24.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 61 last_clean_interval [27,60) [v2:127.0.0.1:6826/905749145,v1:127.0.0.1:6827/905749145] [v2:127.0.0.1:6828/905749145,v1:127.0.0.1:6829/905749145] exists,up 2abc883d-9507-4d62-866e-994d63a5b3cc 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_object_attr_read_error: wait_for_clean 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:41:24.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:41:24.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:41:24.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:41:24.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:41:24.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:41:24.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:41:24.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:41:24.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:41:24.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:41:24.486 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:41:24.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:24.719 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:41:24.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803787 2026-02-25T16:41:24.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803787 2026-02-25T16:41:24.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787' 2026-02-25T16:41:24.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:24.784 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:41:24.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-25T16:41:24.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-25T16:41:24.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858' 2026-02-25T16:41:24.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:24.850 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:41:24.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411338 2026-02-25T16:41:24.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411338 2026-02-25T16:41:24.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411338' 2026-02-25T16:41:24.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:24.913 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:41:24.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=274877906947 2026-02-25T16:41:24.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 274877906947 2026-02-25T16:41:24.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411338 3-274877906947' 2026-02-25T16:41:24.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:24.975 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:41:25.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855368 2026-02-25T16:41:25.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855368 2026-02-25T16:41:25.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411338 3-274877906947 4-150323855368' 2026-02-25T16:41:25.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:25.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:41:25.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561032 2026-02-25T16:41:25.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561032 2026-02-25T16:41:25.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411338 3-274877906947 4-150323855368 5-188978561032' 2026-02-25T16:41:25.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:25.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:41:25.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=227633266695 2026-02-25T16:41:25.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 227633266695 2026-02-25T16:41:25.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-77309411338 3-274877906947 4-150323855368 5-188978561032 6-227633266695' 2026-02-25T16:41:25.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:25.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803787 2026-02-25T16:41:25.168 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:25.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:41:25.169 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803787 2026-02-25T16:41:25.169 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:25.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803787 2026-02-25T16:41:25.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2026-02-25T16:41:25.170 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803787 2026-02-25T16:41:25.170 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:41:25.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803787 2026-02-25T16:41:25.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:41:26.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:41:26.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:41:26.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803787 2026-02-25T16:41:26.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:26.622 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574858 2026-02-25T16:41:26.622 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:26.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:41:26.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574858 2026-02-25T16:41:26.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:26.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574858 2026-02-25T16:41:26.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574858' 2026-02-25T16:41:26.625 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574858 2026-02-25T16:41:26.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:41:26.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574858 -lt 55834574858 2026-02-25T16:41:26.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:26.849 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411338 2026-02-25T16:41:26.849 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:26.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:41:26.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411338 2026-02-25T16:41:26.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:26.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411338 2026-02-25T16:41:26.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411338' 2026-02-25T16:41:26.852 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411338 2026-02-25T16:41:26.852 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:41:27.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411338 -lt 77309411338 2026-02-25T16:41:27.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:27.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-274877906947 2026-02-25T16:41:27.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:27.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:41:27.083 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-274877906947 2026-02-25T16:41:27.083 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:27.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=274877906947 2026-02-25T16:41:27.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 274877906947' 2026-02-25T16:41:27.084 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 274877906947 2026-02-25T16:41:27.085 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:41:27.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 274877906947 -lt 274877906947 2026-02-25T16:41:27.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:27.306 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855368 2026-02-25T16:41:27.306 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:27.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:41:27.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855368 2026-02-25T16:41:27.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:27.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855368 2026-02-25T16:41:27.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855368' 2026-02-25T16:41:27.308 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855368 2026-02-25T16:41:27.308 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:41:27.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855368 -lt 150323855368 2026-02-25T16:41:27.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:27.533 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-188978561032 2026-02-25T16:41:27.533 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:27.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:41:27.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-188978561032 2026-02-25T16:41:27.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:27.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561032 2026-02-25T16:41:27.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 188978561032' 2026-02-25T16:41:27.536 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 188978561032 2026-02-25T16:41:27.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:41:27.766 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561032 -lt 188978561032 2026-02-25T16:41:27.766 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:27.766 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-227633266695 2026-02-25T16:41:27.766 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:27.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:41:27.767 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-227633266695 2026-02-25T16:41:27.768 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:27.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=227633266695 2026-02-25T16:41:27.769 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 227633266695' 2026-02-25T16:41:27.769 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 227633266695 2026-02-25T16:41:27.769 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:41:27.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 227633266695 -lt 227633266695 2026-02-25T16:41:27.996 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:41:27.996 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:41:27.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:41:28.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:41:28.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:41:28.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:41:28.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:41:28.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:41:28.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:41:28.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:41:28.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:41:28.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:403: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:41:28.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:41:28.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:41:28.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-25T16:41:28.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:41:28.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:41:28.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-25T16:41:28.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:41:28.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:41:28.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:41:28.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:41:28.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:41:29.128 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:41:29.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:41:29.415 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:41:29.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:41:29.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:41:29.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:41:29.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:41:29.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:41:29.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:41:29.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:41:29.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:41:29.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:41:29.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:41:29.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:41:29.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:41:29.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:41:29.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:41:29.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:41:29.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:41:29.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:41:29.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:41:29.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:41:29.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:41:29.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:41:29.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:41:29.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:41:29.553 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:41:29.553 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:29.554 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:29.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:41:29.556 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:41:29.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:41:29.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:41:29.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:41:29.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:41:29.558 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:41:29.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:41:29.558 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:41:29.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:41:29.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:41:29.559 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:41:29.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:41:29.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:41:29.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:41:29.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:41:29.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:41:29.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:41:29.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:41:29.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:41:29.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:29.564 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:29.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:41:29.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:41:29.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:41:29.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:41:29.566 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:41:29.566 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:29.566 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:29.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:41:29.567 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:41:29.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:41:29.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:41:29.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:41:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:41:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:41:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:41:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:41:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:41:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:41:29.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:41:29.586 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:41:29.587 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:29.587 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:29.587 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:29.587 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:29.587 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:29.587 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:29.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:41:29.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:41:29.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:41:29.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:29.612 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:41:29.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:41:29.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:41:29.656 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:41:29.656 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:41:29.656 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:41:29.657 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:41:29.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:41:29.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:41:29.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:41:29.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:41:29.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:41:29.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:41:29.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:41:29.798 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:41:29.798 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:29.799 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:29.799 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:29.799 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:29.799 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:29.799 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:29.800 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:41:29.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:41:29.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:41:29.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:41:29.911 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:41:29.917 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:41:30.920 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:41:30.920 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:41:30.921 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:41:30.921 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:41:30.921 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:30.921 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:30.921 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:41:30.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:41:30.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:41:30.960 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:41:30.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:41:30.965 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:41:29.602+0000 7efee800cc00 0 load: jerasure load: lrc 2026-02-25T16:41:30.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2026-02-25T16:41:30.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2026-02-25T16:41:30.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:439: TEST_ec_recovery_multiple_errors: local objname=myobject 2026-02-25T16:41:30.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: setup_osds 9 2026-02-25T16:41:30.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2026-02-25T16:41:30.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:41:30.966 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2026-02-25T16:41:30.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:30.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:30.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:30.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:30.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:30.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:41:30.970 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:30.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e313b3f3-3f38-411b-b069-d5ad86b088c7 2026-02-25T16:41:30.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e313b3f3-3f38-411b-b069-d5ad86b088c7' 2026-02-25T16:41:30.971 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 e313b3f3-3f38-411b-b069-d5ad86b088c7 2026-02-25T16:41:30.971 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:30.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6Jp9pv1ppOhAAKwGqgR18dTqJ/5bn1a8JcA== 2026-02-25T16:41:30.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6Jp9pv1ppOhAAKwGqgR18dTqJ/5bn1a8JcA=="}' 2026-02-25T16:41:30.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e313b3f3-3f38-411b-b069-d5ad86b088c7 -i td/test-erasure-eio/0/new.json 2026-02-25T16:41:31.083 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:31.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:41:31.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6Jp9pv1ppOhAAKwGqgR18dTqJ/5bn1a8JcA== --osd-uuid e313b3f3-3f38-411b-b069-d5ad86b088c7 2026-02-25T16:41:31.106 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.104+0000 7f5500866780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:31.107 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.106+0000 7f5500866780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:31.108 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.106+0000 7f5500866780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:31.109 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.107+0000 7f5500866780 -1 bdev(0x3ad6c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:31.109 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.107+0000 7f5500866780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:41:31.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:41:31.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:31.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:41:31.552 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:41:31.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:31.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:41:31.655 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:41:31.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:31.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:31.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:31.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:31.669 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.668+0000 7f3ca742e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:31.671 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.669+0000 7f3ca742e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:31.672 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.670+0000 7f3ca742e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:31.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:41:31.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:31.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:41:31.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:31.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:31.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:31.751 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:31.751 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:31.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:31.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:41:31.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:31.879 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:31.878+0000 7f3ca742e780 -1 Falling back to public interface 2026-02-25T16:41:32.150 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:32.148+0000 7f3ca742e780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:41:32.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:32.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:32.847 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:32.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:32.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:32.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:41:33.070 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/831325168,v1:127.0.0.1:6803/831325168] [v2:127.0.0.1:6804/831325168,v1:127.0.0.1:6805/831325168] exists,up e313b3f3-3f38-411b-b069-d5ad86b088c7 2026-02-25T16:41:33.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:33.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:33.075 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:33.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:33.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:33.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:41:33.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:33.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d8c2af4-4059-481e-90dd-d728bf2e7274 2026-02-25T16:41:33.077 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 5d8c2af4-4059-481e-90dd-d728bf2e7274 2026-02-25T16:41:33.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5d8c2af4-4059-481e-90dd-d728bf2e7274' 2026-02-25T16:41:33.078 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:33.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9Jp9psrHmBBAABOxaQ1PdMpFP6VSGQgmp8A== 2026-02-25T16:41:33.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9Jp9psrHmBBAABOxaQ1PdMpFP6VSGQgmp8A=="}' 2026-02-25T16:41:33.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d8c2af4-4059-481e-90dd-d728bf2e7274 -i td/test-erasure-eio/1/new.json 2026-02-25T16:41:33.310 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:33.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:41:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9Jp9psrHmBBAABOxaQ1PdMpFP6VSGQgmp8A== --osd-uuid 5d8c2af4-4059-481e-90dd-d728bf2e7274 2026-02-25T16:41:33.333 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:33.332+0000 7f85a6662780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:33.335 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:33.333+0000 7f85a6662780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:33.336 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:33.334+0000 7f85a6662780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:33.336 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:33.334+0000 7f85a6662780 -1 bdev(0x37bac00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:33.336 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:33.334+0000 7f85a6662780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:41:33.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:41:33.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:33.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:41:33.682 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:41:33.683 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:33.999 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:41:34.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:41:34.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:34.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:34.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:34.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:34.013 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:34.011+0000 7f8f6382f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:34.014 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:34.012+0000 7f8f6382f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:34.015 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:34.013+0000 7f8f6382f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:34.208 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:34.206+0000 7f8f6382f780 -1 Falling back to public interface 2026-02-25T16:41:34.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:41:34.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:34.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:41:34.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:34.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:34.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:34.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:34.228 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:34.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:34.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:41:34.453 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:34.588 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:34.586+0000 7f8f6382f780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:41:35.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:35.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:35.455 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:35.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:35.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:35.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:41:35.680 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2480900943,v1:127.0.0.1:6811/2480900943] [v2:127.0.0.1:6812/2480900943,v1:127.0.0.1:6813/2480900943] exists,up 5d8c2af4-4059-481e-90dd-d728bf2e7274 2026-02-25T16:41:35.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:41:35.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:35.686 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:35.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:35.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:35.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:35.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:41:35.688 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:35.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d2eab16-e45a-4875-a7c4-c3ecee15a514 2026-02-25T16:41:35.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8d2eab16-e45a-4875-a7c4-c3ecee15a514' 2026-02-25T16:41:35.688 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 8d2eab16-e45a-4875-a7c4-c3ecee15a514 2026-02-25T16:41:35.689 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:35.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA/Jp9pARtKKRAAiwJwAglQZKX3dmCngeaW5g== 2026-02-25T16:41:35.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA/Jp9pARtKKRAAiwJwAglQZKX3dmCngeaW5g=="}' 2026-02-25T16:41:35.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d2eab16-e45a-4875-a7c4-c3ecee15a514 -i td/test-erasure-eio/2/new.json 2026-02-25T16:41:35.925 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:41:35.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:41:35.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/Jp9pARtKKRAAiwJwAglQZKX3dmCngeaW5g== --osd-uuid 8d2eab16-e45a-4875-a7c4-c3ecee15a514 2026-02-25T16:41:35.948 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:35.947+0000 7f688360e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:35.950 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:35.948+0000 7f688360e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:35.951 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:35.949+0000 7f688360e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:35.951 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:35.950+0000 7f688360e780 -1 bdev(0x2af4c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:35.951 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:35.950+0000 7f688360e780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:41:36.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:41:36.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:36.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:41:36.293 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:41:36.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:36.606 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:41:36.607 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:41:36.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:36.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:36.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:36.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:36.621 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:36.620+0000 7f818544b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:36.623 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:36.621+0000 7f818544b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:36.624 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:36.622+0000 7f818544b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:36.752 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:36.751+0000 7f818544b780 -1 Falling back to public interface 2026-02-25T16:41:36.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:41:36.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:36.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:41:36.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:36.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:36.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:36.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:36.833 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:36.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:36.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:41:37.055 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:37.121 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:37.120+0000 7f818544b780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:41:38.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:38.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:38.057 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:38.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:38.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:38.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:41:38.280 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2661517429,v1:127.0.0.1:6819/2661517429] [v2:127.0.0.1:6820/2661517429,v1:127.0.0.1:6821/2661517429] exists,up 8d2eab16-e45a-4875-a7c4-c3ecee15a514 2026-02-25T16:41:38.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:38.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:38.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:38.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:38.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:41:38.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:38.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:38.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:38.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:41:38.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:38.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0eebd835-5be7-463b-a77f-81a0d251017c 2026-02-25T16:41:38.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0eebd835-5be7-463b-a77f-81a0d251017c' 2026-02-25T16:41:38.289 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 0eebd835-5be7-463b-a77f-81a0d251017c 2026-02-25T16:41:38.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:38.293 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCJp9pCRRtERAAnFOhljN+vofXkxUmxLGdbw== 2026-02-25T16:41:38.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCJp9pCRRtERAAnFOhljN+vofXkxUmxLGdbw=="}' 2026-02-25T16:41:38.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0eebd835-5be7-463b-a77f-81a0d251017c -i td/test-erasure-eio/3/new.json 2026-02-25T16:41:38.522 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:41:38.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:41:38.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCJp9pCRRtERAAnFOhljN+vofXkxUmxLGdbw== --osd-uuid 0eebd835-5be7-463b-a77f-81a0d251017c 2026-02-25T16:41:38.547 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:38.545+0000 7fa035c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:38.548 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:38.547+0000 7fa035c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:38.549 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:38.548+0000 7fa035c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:38.550 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:38.548+0000 7fa035c1c780 -1 bdev(0x341cc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:38.550 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:38.548+0000 7fa035c1c780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:41:38.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:41:38.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:38.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:41:38.910 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:41:38.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:39.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:41:39.229 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:41:39.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:39.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:39.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:39.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:39.243 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:39.242+0000 7f5c0ee5d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:39.245 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:39.243+0000 7f5c0ee5d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:39.245 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:39.244+0000 7f5c0ee5d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:39.372 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:39.371+0000 7f5c0ee5d780 -1 Falling back to public interface 2026-02-25T16:41:39.453 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:41:39.453 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:39.453 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:41:39.453 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:39.453 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:39.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:39.454 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:39.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:39.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:39.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:41:39.670 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:39.668+0000 7f5c0ee5d780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:41:39.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:40.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:40.683 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:40.683 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:40.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:40.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:40.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:41:40.907 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/171272621,v1:127.0.0.1:6827/171272621] [v2:127.0.0.1:6828/171272621,v1:127.0.0.1:6829/171272621] exists,up 0eebd835-5be7-463b-a77f-81a0d251017c 2026-02-25T16:41:40.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:40.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:40.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:40.912 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:40.913 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:40.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:40.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:40.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:40.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T16:41:40.915 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:40.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bf43c752-b225-4335-a22f-12a84db8cda5 2026-02-25T16:41:40.915 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 bf43c752-b225-4335-a22f-12a84db8cda5' 2026-02-25T16:41:40.915 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 bf43c752-b225-4335-a22f-12a84db8cda5 2026-02-25T16:41:40.916 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:40.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEJp9pdh7TNhAAjcvPGZJvqZQUaFLJMbSuQw== 2026-02-25T16:41:40.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEJp9pdh7TNhAAjcvPGZJvqZQUaFLJMbSuQw=="}' 2026-02-25T16:41:40.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bf43c752-b225-4335-a22f-12a84db8cda5 -i td/test-erasure-eio/4/new.json 2026-02-25T16:41:41.150 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:41:41.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T16:41:41.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEJp9pdh7TNhAAjcvPGZJvqZQUaFLJMbSuQw== --osd-uuid bf43c752-b225-4335-a22f-12a84db8cda5 2026-02-25T16:41:41.174 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.173+0000 7fda53614780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:41.176 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.174+0000 7fda53614780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:41.177 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.175+0000 7fda53614780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:41.177 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.176+0000 7fda53614780 -1 bdev(0x2d2cc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:41.177 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.176+0000 7fda53614780 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T16:41:41.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T16:41:41.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:41.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:41:41.533 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:41:41.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:41.847 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:41:41.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:41:41.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:41.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:41.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:41.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:41.862 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.860+0000 7f2b9f7dd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:41.863 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.862+0000 7f2b9f7dd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:41.864 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:41.863+0000 7f2b9f7dd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:42.013 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:42.012+0000 7f2b9f7dd780 -1 Falling back to public interface 2026-02-25T16:41:42.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:41:42.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:42.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:41:42.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:42.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:42.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:42.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:42.078 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:42.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:42.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:41:42.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:42.332 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:42.330+0000 7f2b9f7dd780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:41:43.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:43.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:43.305 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:43.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:43.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:43.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 33 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/298084303,v1:127.0.0.1:6835/298084303] [v2:127.0.0.1:6836/298084303,v1:127.0.0.1:6837/298084303] exists,up bf43c752-b225-4335-a22f-12a84db8cda5 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:43.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:43.537 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T16:41:43.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:43.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fa2289d6-a837-4a0d-891a-5f287cabaf04 2026-02-25T16:41:43.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 fa2289d6-a837-4a0d-891a-5f287cabaf04' 2026-02-25T16:41:43.539 INFO:tasks.workunit.client.0.trial129.stdout:add osd5 fa2289d6-a837-4a0d-891a-5f287cabaf04 2026-02-25T16:41:43.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:43.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHJp9p2JusIBAANv3xuPyrhp0wlz9pDKl1zw== 2026-02-25T16:41:43.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHJp9p2JusIBAANv3xuPyrhp0wlz9pDKl1zw=="}' 2026-02-25T16:41:43.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fa2289d6-a837-4a0d-891a-5f287cabaf04 -i td/test-erasure-eio/5/new.json 2026-02-25T16:41:43.777 INFO:tasks.workunit.client.0.trial129.stdout:5 2026-02-25T16:41:43.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T16:41:43.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHJp9p2JusIBAANv3xuPyrhp0wlz9pDKl1zw== --osd-uuid fa2289d6-a837-4a0d-891a-5f287cabaf04 2026-02-25T16:41:43.801 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:43.799+0000 7f6615238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:43.802 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:43.800+0000 7f6615238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:43.803 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:43.801+0000 7f6615238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:43.803 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:43.802+0000 7f6615238780 -1 bdev(0x3030c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:43.803 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:43.802+0000 7f6615238780 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T16:41:44.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T16:41:44.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:44.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T16:41:44.141 INFO:tasks.workunit.client.0.trial129.stdout:adding osd5 key to auth repository 2026-02-25T16:41:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:44.456 INFO:tasks.workunit.client.0.trial129.stdout:start osd.5 2026-02-25T16:41:44.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T16:41:44.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:44.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:44.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:44.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:44.470 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:44.468+0000 7fc524a51780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:44.471 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:44.470+0000 7fc524a51780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:44.472 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:44.471+0000 7fc524a51780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:44.600 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:44.599+0000 7fc524a51780 -1 Falling back to public interface 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:44.681 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:44.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:44.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:41:44.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:44.951 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:44.950+0000 7fc524a51780 -1 osd.5 0 log_to_monitors true 2026-02-25T16:41:45.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:45.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:45.912 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:45.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:45.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:45.913 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:41:46.141 INFO:tasks.workunit.client.0.trial129.stdout:osd.5 up in weight 1 up_from 42 up_thru 43 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3053456113,v1:127.0.0.1:6843/3053456113] [v2:127.0.0.1:6844/3053456113,v1:127.0.0.1:6845/3053456113] exists,up fa2289d6-a837-4a0d-891a-5f287cabaf04 2026-02-25T16:41:46.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:46.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:46.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:46.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:46.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T16:41:46.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:46.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:46.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T16:41:46.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:46.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T16:41:46.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:46.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:46.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:46.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:46.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T16:41:46.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T16:41:46.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:46.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:46.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:46.148 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:46.148 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:46.148 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:46.149 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:46.149 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:46.149 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:46.149 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:46.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:46.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:46.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:46.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:46.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:46.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:46.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:46.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:46.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:46.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:46.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:46.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:46.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T16:41:46.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:46.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=496ebfd5-5f70-4d98-af37-f389160ecad1 2026-02-25T16:41:46.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 496ebfd5-5f70-4d98-af37-f389160ecad1' 2026-02-25T16:41:46.152 INFO:tasks.workunit.client.0.trial129.stdout:add osd6 496ebfd5-5f70-4d98-af37-f389160ecad1 2026-02-25T16:41:46.153 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:46.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBKJp9pFbEnCRAAeVPKhTFbgbyYvbNHb/BY7g== 2026-02-25T16:41:46.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBKJp9pFbEnCRAAeVPKhTFbgbyYvbNHb/BY7g=="}' 2026-02-25T16:41:46.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 496ebfd5-5f70-4d98-af37-f389160ecad1 -i td/test-erasure-eio/6/new.json 2026-02-25T16:41:46.383 INFO:tasks.workunit.client.0.trial129.stdout:6 2026-02-25T16:41:46.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T16:41:46.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKJp9pFbEnCRAAeVPKhTFbgbyYvbNHb/BY7g== --osd-uuid 496ebfd5-5f70-4d98-af37-f389160ecad1 2026-02-25T16:41:46.406 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:46.405+0000 7fc1a5c14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:46.408 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:46.406+0000 7fc1a5c14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:46.409 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:46.407+0000 7fc1a5c14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:46.409 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:46.408+0000 7fc1a5c14780 -1 bdev(0x28b2c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:46.409 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:46.408+0000 7fc1a5c14780 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T16:41:46.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T16:41:46.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:46.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T16:41:46.756 INFO:tasks.workunit.client.0.trial129.stdout:adding osd6 key to auth repository 2026-02-25T16:41:46.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:47.071 INFO:tasks.workunit.client.0.trial129.stdout:start osd.6 2026-02-25T16:41:47.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T16:41:47.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:47.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:47.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:47.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:47.086 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:47.084+0000 7ff90720d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:47.087 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:47.085+0000 7ff90720d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:47.088 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:47.086+0000 7ff90720d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:47.221 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:47.220+0000 7ff90720d780 -1 Falling back to public interface 2026-02-25T16:41:47.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T16:41:47.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:47.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T16:41:47.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:47.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:47.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:47.301 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:47.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:47.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:47.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:41:47.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:47.551 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:47.549+0000 7ff90720d780 -1 osd.6 0 log_to_monitors true 2026-02-25T16:41:48.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:48.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:48.528 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:48.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:48.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:48.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:41:48.751 INFO:tasks.workunit.client.0.trial129.stdout:osd.6 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3831218880,v1:127.0.0.1:6851/3831218880] [v2:127.0.0.1:6852/3831218880,v1:127.0.0.1:6853/3831218880] exists,up 496ebfd5-5f70-4d98-af37-f389160ecad1 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:48.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/7 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:48.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:48.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:48.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:48.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:48.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:48.757 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:48.757 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:48.758 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:48.758 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:48.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:48.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:48.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:48.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:48.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:48.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:48.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:48.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:48.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:48.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:48.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:48.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:48.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/7 2026-02-25T16:41:48.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:48.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0b4c0e0f-c6ab-4a3d-afc8-213e690bf60c 2026-02-25T16:41:48.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 0b4c0e0f-c6ab-4a3d-afc8-213e690bf60c' 2026-02-25T16:41:48.761 INFO:tasks.workunit.client.0.trial129.stdout:add osd7 0b4c0e0f-c6ab-4a3d-afc8-213e690bf60c 2026-02-25T16:41:48.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:48.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBMJp9pAKR3LRAAWEVzraTMC2p4ouqODRQskA== 2026-02-25T16:41:48.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBMJp9pAKR3LRAAWEVzraTMC2p4ouqODRQskA=="}' 2026-02-25T16:41:48.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0b4c0e0f-c6ab-4a3d-afc8-213e690bf60c -i td/test-erasure-eio/7/new.json 2026-02-25T16:41:48.990 INFO:tasks.workunit.client.0.trial129.stdout:7 2026-02-25T16:41:48.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/7/new.json 2026-02-25T16:41:48.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMJp9pAKR3LRAAWEVzraTMC2p4ouqODRQskA== --osd-uuid 0b4c0e0f-c6ab-4a3d-afc8-213e690bf60c 2026-02-25T16:41:49.013 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.011+0000 7f81d3044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:49.014 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.013+0000 7f81d3044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:49.015 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.014+0000 7f81d3044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:49.016 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.014+0000 7f81d3044780 -1 bdev(0x27c8c00 td/test-erasure-eio/7/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:49.016 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.014+0000 7f81d3044780 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2026-02-25T16:41:49.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2026-02-25T16:41:49.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:49.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-25T16:41:49.385 INFO:tasks.workunit.client.0.trial129.stdout:adding osd7 key to auth repository 2026-02-25T16:41:49.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:49.701 INFO:tasks.workunit.client.0.trial129.stdout:start osd.7 2026-02-25T16:41:49.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-25T16:41:49.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:49.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:49.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:49.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:49.716 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.715+0000 7f3d65e2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:49.718 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.716+0000 7f3d65e2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:49.718 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.717+0000 7f3d65e2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:49.860 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:49.858+0000 7f3d65e2e780 -1 Falling back to public interface 2026-02-25T16:41:49.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-25T16:41:49.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:49.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-25T16:41:49.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:49.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:49.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:49.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:49.927 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:49.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:49.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T16:41:50.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:50.176 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:50.174+0000 7f3d65e2e780 -1 osd.7 0 log_to_monitors true 2026-02-25T16:41:51.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:51.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:51.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:51.154 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:51.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:51.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-25T16:41:51.381 INFO:tasks.workunit.client.0.trial129.stdout:osd.7 up in weight 1 up_from 60 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/894518649,v1:127.0.0.1:6859/894518649] [v2:127.0.0.1:6860/894518649,v1:127.0.0.1:6861/894518649] exists,up 0b4c0e0f-c6ab-4a3d-afc8-213e690bf60c 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:41:51.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/8 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:41:51.387 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:41:51.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/8 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ad14a303-ae2f-4090-a58f-aa7fe2d89031 2026-02-25T16:41:51.389 INFO:tasks.workunit.client.0.trial129.stdout:add osd8 ad14a303-ae2f-4090-a58f-aa7fe2d89031 2026-02-25T16:41:51.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 ad14a303-ae2f-4090-a58f-aa7fe2d89031' 2026-02-25T16:41:51.390 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:41:51.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBPJp9ppIN2FxAADidTPmwPvgM843vEIY2kJA== 2026-02-25T16:41:51.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBPJp9ppIN2FxAADidTPmwPvgM843vEIY2kJA=="}' 2026-02-25T16:41:51.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ad14a303-ae2f-4090-a58f-aa7fe2d89031 -i td/test-erasure-eio/8/new.json 2026-02-25T16:41:51.621 INFO:tasks.workunit.client.0.trial129.stdout:8 2026-02-25T16:41:51.628 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/8/new.json 2026-02-25T16:41:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPJp9ppIN2FxAADidTPmwPvgM843vEIY2kJA== --osd-uuid ad14a303-ae2f-4090-a58f-aa7fe2d89031 2026-02-25T16:41:51.643 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:51.641+0000 7f2dbe635780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:51.645 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:51.643+0000 7f2dbe635780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:51.645 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:51.644+0000 7f2dbe635780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:51.646 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:51.644+0000 7f2dbe635780 -1 bdev(0x35aec00 td/test-erasure-eio/8/block) open stat got: (1) Operation not permitted 2026-02-25T16:41:51.646 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:51.644+0000 7f2dbe635780 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2026-02-25T16:41:52.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2026-02-25T16:41:52.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:41:52.012 INFO:tasks.workunit.client.0.trial129.stdout:adding osd8 key to auth repository 2026-02-25T16:41:52.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-25T16:41:52.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:41:52.329 INFO:tasks.workunit.client.0.trial129.stdout:start osd.8 2026-02-25T16:41:52.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-25T16:41:52.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:41:52.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:41:52.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:41:52.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:41:52.342 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:52.341+0000 7f1c2f1d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:52.344 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:52.342+0000 7f1c2f1d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:52.344 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:52.343+0000 7f1c2f1d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:41:52.466 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:52.464+0000 7f1c2f1d8780 -1 Falling back to public interface 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:52.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T16:41:52.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:41:52.793 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:41:52.791+0000 7f1c2f1d8780 -1 osd.8 0 log_to_monitors true 2026-02-25T16:41:53.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:41:53.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:41:53.783 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:41:53.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:41:53.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:41:53.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-25T16:41:54.010 INFO:tasks.workunit.client.0.trial129.stdout:osd.8 up in weight 1 up_from 67 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2432402309,v1:127.0.0.1:6867/2432402309] [v2:127.0.0.1:6868/2432402309,v1:127.0.0.1:6869/2432402309] exists,up ad14a303-ae2f-4090-a58f-aa7fe2d89031 2026-02-25T16:41:54.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:41:54.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:41:54.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:41:54.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:41:54.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:41:54.056 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:41:54.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:41:54.062 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:41:31.880+0000 7f3ca742e780 0 load: jerasure load: lrc 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:444: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:41:54.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2026-02-25T16:41:54.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:41:54.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:41:54.729 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:41:54.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:41:55.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:41:55.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:41:55.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:41:55.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:41:55.740 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:41:55.740 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:41:55.740 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:41:55.740 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:41:55.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:41:55.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:41:55.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:41:55.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:41:55.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:41:55.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:41:55.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:41:55.824 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:41:56.049 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:8' 2026-02-25T16:41:56.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:41:56.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.051 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:41:56.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803783 2026-02-25T16:41:56.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803783 2026-02-25T16:41:56.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783' 2026-02-25T16:41:56.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:41:56.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607559 2026-02-25T16:41:56.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607559 2026-02-25T16:41:56.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559' 2026-02-25T16:41:56.183 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.183 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:41:56.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-25T16:41:56.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-25T16:41:56.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559 2-77309411334' 2026-02-25T16:41:56.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.247 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:41:56.312 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182406 2026-02-25T16:41:56.312 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182406 2026-02-25T16:41:56.312 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559 2-77309411334 3-107374182406' 2026-02-25T16:41:56.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:41:56.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920774 2026-02-25T16:41:56.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920774 2026-02-25T16:41:56.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559 2-77309411334 3-107374182406 4-141733920774' 2026-02-25T16:41:56.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.377 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:41:56.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626437 2026-02-25T16:41:56.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626437 2026-02-25T16:41:56.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559 2-77309411334 3-107374182406 4-141733920774 5-180388626437' 2026-02-25T16:41:56.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.440 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:41:56.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364804 2026-02-25T16:41:56.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364804 2026-02-25T16:41:56.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559 2-77309411334 3-107374182406 4-141733920774 5-180388626437 6-214748364804' 2026-02-25T16:41:56.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.503 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:41:56.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037764 2026-02-25T16:41:56.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037764 2026-02-25T16:41:56.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559 2-77309411334 3-107374182406 4-141733920774 5-180388626437 6-214748364804 7-257698037764' 2026-02-25T16:41:56.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:41:56.565 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:41:56.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808835 2026-02-25T16:41:56.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808835 2026-02-25T16:41:56.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607559 2-77309411334 3-107374182406 4-141733920774 5-180388626437 6-214748364804 7-257698037764 8-287762808835' 2026-02-25T16:41:56.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:56.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803783 2026-02-25T16:41:56.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:56.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:41:56.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803783 2026-02-25T16:41:56.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:56.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803783 2026-02-25T16:41:56.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803783' 2026-02-25T16:41:56.634 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803783 2026-02-25T16:41:56.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:41:56.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803783 2026-02-25T16:41:56.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:41:57.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:41:57.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:41:58.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803783 2026-02-25T16:41:58.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:58.088 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607559 2026-02-25T16:41:58.088 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:58.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:41:58.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607559 2026-02-25T16:41:58.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:58.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607559 2026-02-25T16:41:58.091 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607559 2026-02-25T16:41:58.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607559' 2026-02-25T16:41:58.091 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:41:58.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607559 -lt 51539607559 2026-02-25T16:41:58.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:58.322 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411334 2026-02-25T16:41:58.322 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:58.322 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:41:58.323 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411334 2026-02-25T16:41:58.323 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:58.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411334 2026-02-25T16:41:58.324 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411334 2026-02-25T16:41:58.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411334' 2026-02-25T16:41:58.324 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:41:58.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411334 -lt 77309411334 2026-02-25T16:41:58.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:58.552 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-107374182406 2026-02-25T16:41:58.552 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:58.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:41:58.553 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-107374182406 2026-02-25T16:41:58.553 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:58.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=107374182406 2026-02-25T16:41:58.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 107374182406' 2026-02-25T16:41:58.555 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 107374182406 2026-02-25T16:41:58.555 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:41:58.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182406 -lt 107374182406 2026-02-25T16:41:58.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:58.782 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-141733920774 2026-02-25T16:41:58.782 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:58.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:41:58.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-141733920774 2026-02-25T16:41:58.784 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:58.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=141733920774 2026-02-25T16:41:58.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 141733920774' 2026-02-25T16:41:58.785 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 141733920774 2026-02-25T16:41:58.785 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:41:59.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 141733920774 -lt 141733920774 2026-02-25T16:41:59.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:59.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-180388626437 2026-02-25T16:41:59.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:59.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:41:59.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-180388626437 2026-02-25T16:41:59.014 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:59.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=180388626437 2026-02-25T16:41:59.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 180388626437' 2026-02-25T16:41:59.015 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 180388626437 2026-02-25T16:41:59.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:41:59.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 180388626437 -lt 180388626437 2026-02-25T16:41:59.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:59.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-214748364804 2026-02-25T16:41:59.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:59.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:41:59.241 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-214748364804 2026-02-25T16:41:59.241 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:59.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=214748364804 2026-02-25T16:41:59.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 214748364804' 2026-02-25T16:41:59.242 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 214748364804 2026-02-25T16:41:59.243 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:41:59.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 214748364804 -lt 214748364804 2026-02-25T16:41:59.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:59.464 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 7-257698037764 2026-02-25T16:41:59.465 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:59.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=7 2026-02-25T16:41:59.466 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-257698037764 2026-02-25T16:41:59.466 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:59.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=257698037764 2026-02-25T16:41:59.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.7 seq 257698037764' 2026-02-25T16:41:59.467 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.7 seq 257698037764 2026-02-25T16:41:59.468 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-25T16:41:59.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 257698037764 -lt 257698037764 2026-02-25T16:41:59.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:41:59.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-287762808835 2026-02-25T16:41:59.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:41:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:41:59.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-287762808835 2026-02-25T16:41:59.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:41:59.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=287762808835 2026-02-25T16:41:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 287762808835' 2026-02-25T16:41:59.697 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.8 seq 287762808835 2026-02-25T16:41:59.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:41:59.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 287762808836 -lt 287762808835 2026-02-25T16:41:59.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:41:59.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:41:59.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:42:00.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:42:00.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:42:00.464 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:42:00.465 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:42:00.465 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:00.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:446: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:00.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:42:00.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:00.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:42:00.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:00.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:42:00.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T16:42:00.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:447: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-25T16:42:00.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:42:00.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T16:42:00.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:42:00.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:42:00.795 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:42:00.795 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:01.020 INFO:tasks.workunit.client.0.trial129.stderr:7' 2026-02-25T16:42:01.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T16:42:01.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T16:42:01.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:42:01.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T16:42:01.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:42:01.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T16:42:01.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:42:01.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T16:42:01.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:42:01.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T16:42:01.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:42:01.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:42:01.024 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:42:01.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:42:01.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:42:01.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:42:01.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-25T16:42:01.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:450: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2026-02-25T16:42:01.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:42:01.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:42:01.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:42:01.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:42:01.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=3 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:42:01.121 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:7' 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=0 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:42:01.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/0/type 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-25T16:42:01.350 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:42:01.351 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:42:01.351 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:01.351 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:01.351 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:01.351 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:42:01.351 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:42:01.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:42:01.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:42:01.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:42:01.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.0 2026-02-25T16:42:01.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:42:01.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:42:01.399 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:01.399 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:01.399 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:01.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:42:01.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:42:01.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=4 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:42:01.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:7' 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=6 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:42:01.678 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/6/type 2026-02-25T16:42:01.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:42:01.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2026-02-25T16:42:01.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:42:01.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=6 2026-02-25T16:42:01.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:42:01.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:42:01.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:42:01.680 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.6 2026-02-25T16:42:01.680 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2026-02-25T16:42:01.680 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2026-02-25T16:42:01.681 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:01.681 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:01.681 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:01.681 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.6.asok 2026-02-25T16:42:01.681 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:42:01.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:42:01.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:42:01.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.6 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.6 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.6 ']' 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.6.asok 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:42:01.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2026-02-25T16:42:01.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2026-02-25T16:42:01.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:42:01.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:42:01.781 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:42:01.781 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:7' 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-25T16:42:02.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_errors: local -a initial_osds 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_errors: local last_osd=7 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:42:02.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:42:02.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:42:02.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_errors: ceph osd down 7 2026-02-25T16:42:02.536 INFO:tasks.workunit.client.0.trial129.stderr:osd.7 is already down. 2026-02-25T16:42:02.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:458: TEST_ec_recovery_multiple_errors: ceph osd out 7 2026-02-25T16:42:02.791 INFO:tasks.workunit.client.0.trial129.stderr:osd.7 is already out. 2026-02-25T16:42:02.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:461: TEST_ec_recovery_multiple_errors: wait_for_clean 2026-02-25T16:42:02.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:42:02.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:42:02.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:42:02.799 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:42:02.800 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:42:02.800 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:42:02.800 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:42:02.800 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:42:02.800 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:42:02.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:42:02.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:42:02.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:42:02.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:42:02.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:42:02.885 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:42:03.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:6 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:7 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:8' 2026-02-25T16:42:03.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:42:03.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:42:03.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803786 2026-02-25T16:42:03.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803786 2026-02-25T16:42:03.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786' 2026-02-25T16:42:03.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:42:03.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607562 2026-02-25T16:42:03.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607562 2026-02-25T16:42:03.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607562' 2026-02-25T16:42:03.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.243 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:42:03.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-25T16:42:03.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-25T16:42:03.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607562 2-77309411337' 2026-02-25T16:42:03.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.308 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:42:03.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182409 2026-02-25T16:42:03.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182409 2026-02-25T16:42:03.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607562 2-77309411337 3-107374182409' 2026-02-25T16:42:03.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.372 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:42:03.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920777 2026-02-25T16:42:03.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920777 2026-02-25T16:42:03.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607562 2-77309411337 3-107374182409 4-141733920777' 2026-02-25T16:42:03.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:42:03.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626440 2026-02-25T16:42:03.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626440 2026-02-25T16:42:03.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607562 2-77309411337 3-107374182409 4-141733920777 5-180388626440' 2026-02-25T16:42:03.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.503 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:42:03.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364807 2026-02-25T16:42:03.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364807 2026-02-25T16:42:03.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607562 2-77309411337 3-107374182409 4-141733920777 5-180388626440 6-214748364807' 2026-02-25T16:42:03.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.565 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-25T16:42:03.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T16:42:03.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T16:42:03.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T16:42:03.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T16:42:03.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:03.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-25T16:42:03.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808838 2026-02-25T16:42:03.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808838 2026-02-25T16:42:03.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607562 2-77309411337 3-107374182409 4-141733920777 5-180388626440 6-214748364807 8-287762808838' 2026-02-25T16:42:03.673 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:03.673 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803786 2026-02-25T16:42:03.673 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:03.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:42:03.674 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803786 2026-02-25T16:42:03.674 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:03.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803786 2026-02-25T16:42:03.675 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803786 2026-02-25T16:42:03.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803786' 2026-02-25T16:42:03.676 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:42:03.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803786 2026-02-25T16:42:03.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:03.899 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607562 2026-02-25T16:42:03.900 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:03.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:42:03.901 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607562 2026-02-25T16:42:03.901 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:03.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607562 2026-02-25T16:42:03.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607562' 2026-02-25T16:42:03.902 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607562 2026-02-25T16:42:03.902 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:42:04.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607562 -lt 51539607562 2026-02-25T16:42:04.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:04.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411337 2026-02-25T16:42:04.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:04.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:42:04.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411337 2026-02-25T16:42:04.128 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:04.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411337 2026-02-25T16:42:04.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411337' 2026-02-25T16:42:04.130 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411337 2026-02-25T16:42:04.130 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:42:04.354 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411338 -lt 77309411337 2026-02-25T16:42:04.354 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:04.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-107374182409 2026-02-25T16:42:04.355 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:04.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:42:04.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-107374182409 2026-02-25T16:42:04.356 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:04.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=107374182409 2026-02-25T16:42:04.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 107374182409' 2026-02-25T16:42:04.358 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 107374182409 2026-02-25T16:42:04.358 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:42:04.583 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182409 -lt 107374182409 2026-02-25T16:42:04.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:04.584 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-141733920777 2026-02-25T16:42:04.584 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:04.585 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:42:04.585 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-141733920777 2026-02-25T16:42:04.586 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:04.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=141733920777 2026-02-25T16:42:04.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 141733920777' 2026-02-25T16:42:04.587 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 141733920777 2026-02-25T16:42:04.587 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:42:04.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 141733920778 -lt 141733920777 2026-02-25T16:42:04.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:04.811 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-180388626440 2026-02-25T16:42:04.811 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:04.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:42:04.812 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-180388626440 2026-02-25T16:42:04.813 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:04.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=180388626440 2026-02-25T16:42:04.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 180388626440' 2026-02-25T16:42:04.814 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 180388626440 2026-02-25T16:42:04.814 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:42:05.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 180388626440 -lt 180388626440 2026-02-25T16:42:05.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:05.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-214748364807 2026-02-25T16:42:05.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:05.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:42:05.041 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-214748364807 2026-02-25T16:42:05.041 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:05.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=214748364807 2026-02-25T16:42:05.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 214748364807' 2026-02-25T16:42:05.043 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 214748364807 2026-02-25T16:42:05.043 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:42:05.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 214748364808 -lt 214748364807 2026-02-25T16:42:05.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:05.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 8-287762808838 2026-02-25T16:42:05.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:05.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=8 2026-02-25T16:42:05.270 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-287762808838 2026-02-25T16:42:05.270 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:05.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=287762808838 2026-02-25T16:42:05.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.8 seq 287762808838' 2026-02-25T16:42:05.271 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.8 seq 287762808838 2026-02-25T16:42:05.271 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-25T16:42:05.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 287762808838 -lt 287762808838 2026-02-25T16:42:05.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:42:05.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:42:05.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:05.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:42:05.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:42:05.815 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:42:05.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:42:05.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:42:05.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:42:05.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:42:05.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:42:06.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:42:06.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:42:06.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:42:06.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:42:06.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:42:06.351 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-25T16:42:06.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:42:06.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:42:06.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:465: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:42:06.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:42:06.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:42:06.640 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:42:06.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:42:06.926 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:42:06.934 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:42:06.934 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:42:06.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:42:06.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:42:06.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:42:06.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:42:06.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:42:06.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:42:06.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:42:07.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:42:07.046 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:42:07.046 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:42:07.047 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:42:07.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:42:07.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:42:07.048 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:42:07.049 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:07.049 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:42:07.049 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:42:07.049 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:07.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:42:07.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:42:07.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:42:07.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:42:07.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:07.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:07.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:42:07.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:42:07.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:42:07.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:42:07.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:42:07.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:42:07.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:42:07.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:42:07.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:42:07.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:07.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:42:07.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:42:07.076 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:07.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:42:07.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:42:07.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:42:07.079 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:42:07.079 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:07.079 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:07.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:42:07.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:42:07.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:42:07.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:42:07.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:42:07.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:07.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:07.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:42:07.083 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:42:07.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:42:07.103 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:42:07.103 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:07.104 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:07.104 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:07.104 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:07.104 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:07.104 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:07.104 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:42:07.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:42:07.129 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:42:07.130 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:42:07.130 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:42:07.130 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:42:07.130 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:42:07.130 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:42:07.130 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:42:07.130 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:42:07.131 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:07.131 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:07.131 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:07.131 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:42:07.132 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:42:07.132 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:42:07.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:42:07.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:42:07.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:42:07.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:42:07.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:42:07.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:42:07.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:42:07.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:42:07.177 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:07.177 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:07.177 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:07.177 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:42:07.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:42:07.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:42:07.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:42:07.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:42:07.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:42:07.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:42:07.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:42:07.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:42:07.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:42:07.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:42:07.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:07.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:07.319 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:07.319 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:07.319 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:07.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:07.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:42:07.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:42:07.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:42:07.334 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:42:07.427 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:42:07.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:42:08.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:42:08.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:42:08.474 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:42:08.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:42:08.479 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:42:07.120+0000 7ff08b83fc00 0 load: jerasure load: lrc 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: local objname=myobject 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:474: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:476: TEST_ec_recovery_multiple_objects: setup_osds 7 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T16:42:08.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:42:08.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:08.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:42:08.485 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:08.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fe6f6c36-9305-4aca-b170-dac175031949 2026-02-25T16:42:08.485 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 fe6f6c36-9305-4aca-b170-dac175031949 2026-02-25T16:42:08.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fe6f6c36-9305-4aca-b170-dac175031949' 2026-02-25T16:42:08.485 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:08.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBgJp9pae94HRAApqlJBKAcdx2PelsLD0hYhQ== 2026-02-25T16:42:08.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBgJp9pae94HRAApqlJBKAcdx2PelsLD0hYhQ=="}' 2026-02-25T16:42:08.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fe6f6c36-9305-4aca-b170-dac175031949 -i td/test-erasure-eio/0/new.json 2026-02-25T16:42:08.599 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:08.606 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:42:08.606 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgJp9pae94HRAApqlJBKAcdx2PelsLD0hYhQ== --osd-uuid fe6f6c36-9305-4aca-b170-dac175031949 2026-02-25T16:42:08.625 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:08.623+0000 7f7e2a1d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:08.626 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:08.625+0000 7f7e2a1d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:08.627 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:08.625+0000 7f7e2a1d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:08.628 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:08.626+0000 7f7e2a1d4780 -1 bdev(0x2fd4c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:08.628 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:08.626+0000 7f7e2a1d4780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:42:09.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:42:09.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:09.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:42:09.019 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:42:09.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:09.121 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:42:09.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:42:09.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:09.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:09.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:09.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:09.138 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:09.136+0000 7f93ace2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:09.139 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:09.138+0000 7f93ace2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:09.140 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:09.139+0000 7f93ace2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:09.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:42:09.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:09.219 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:42:09.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:09.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:09.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:09.220 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:09.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:09.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:09.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:42:09.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:09.346 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:09.344+0000 7f93ace2f780 -1 Falling back to public interface 2026-02-25T16:42:09.609 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:09.608+0000 7f93ace2f780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:42:10.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:10.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:10.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:10.319 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:10.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:10.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:42:10.546 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2038481920,v1:127.0.0.1:6803/2038481920] [v2:127.0.0.1:6804/2038481920,v1:127.0.0.1:6805/2038481920] exists,up fe6f6c36-9305-4aca-b170-dac175031949 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:10.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:10.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:10.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:10.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:10.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:10.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:10.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:42:10.549 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:10.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=01597005-ff6c-4b55-9eff-dfcf49a50bad 2026-02-25T16:42:10.550 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 01597005-ff6c-4b55-9eff-dfcf49a50bad 2026-02-25T16:42:10.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 01597005-ff6c-4b55-9eff-dfcf49a50bad' 2026-02-25T16:42:10.550 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBiJp9pg7xuIRAABhjZ84awYeiMyCLqfBXfNA== 2026-02-25T16:42:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBiJp9pg7xuIRAABhjZ84awYeiMyCLqfBXfNA=="}' 2026-02-25T16:42:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 01597005-ff6c-4b55-9eff-dfcf49a50bad -i td/test-erasure-eio/1/new.json 2026-02-25T16:42:10.792 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:10.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:42:10.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiJp9pg7xuIRAABhjZ84awYeiMyCLqfBXfNA== --osd-uuid 01597005-ff6c-4b55-9eff-dfcf49a50bad 2026-02-25T16:42:10.818 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:10.816+0000 7effa1668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:10.819 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:10.818+0000 7effa1668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:10.820 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:10.819+0000 7effa1668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:10.821 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:10.819+0000 7effa1668780 -1 bdev(0x397ec00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:10.821 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:10.819+0000 7effa1668780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:42:11.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:42:11.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:11.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:42:11.160 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:42:11.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:11.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:42:11.478 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:42:11.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:11.494 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:11.493+0000 7f350ae28780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:11.496 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:11.494+0000 7f350ae28780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:11.496 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:11.495+0000 7f350ae28780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:11.635 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:11.633+0000 7f350ae28780 -1 Falling back to public interface 2026-02-25T16:42:11.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:42:11.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:11.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:42:11.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:11.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:11.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:11.704 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:11.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:11.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:11.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:42:11.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:11.955 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:11.953+0000 7f350ae28780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:42:12.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:12.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:12.932 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:12.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:12.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:12.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:42:13.165 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3634649096,v1:127.0.0.1:6811/3634649096] [v2:127.0.0.1:6812/3634649096,v1:127.0.0.1:6813/3634649096] exists,up 01597005-ff6c-4b55-9eff-dfcf49a50bad 2026-02-25T16:42:13.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:13.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:13.170 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:13.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:13.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:13.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:42:13.172 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:13.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b1251d59-9fcf-4e13-812a-73dfdb7a08aa 2026-02-25T16:42:13.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b1251d59-9fcf-4e13-812a-73dfdb7a08aa' 2026-02-25T16:42:13.172 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 b1251d59-9fcf-4e13-812a-73dfdb7a08aa 2026-02-25T16:42:13.173 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:13.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBlJp9pPtqaChAAD7LpQn8/rRCcGWW4i4P+pg== 2026-02-25T16:42:13.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBlJp9pPtqaChAAD7LpQn8/rRCcGWW4i4P+pg=="}' 2026-02-25T16:42:13.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b1251d59-9fcf-4e13-812a-73dfdb7a08aa -i td/test-erasure-eio/2/new.json 2026-02-25T16:42:13.414 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:42:13.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:42:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlJp9pPtqaChAAD7LpQn8/rRCcGWW4i4P+pg== --osd-uuid b1251d59-9fcf-4e13-812a-73dfdb7a08aa 2026-02-25T16:42:13.440 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:13.439+0000 7ff74b028780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:13.442 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:13.440+0000 7ff74b028780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:13.442 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:13.441+0000 7ff74b028780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:13.443 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:13.441+0000 7ff74b028780 -1 bdev(0x3fccc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:13.443 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:13.441+0000 7ff74b028780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:42:13.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:42:13.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:13.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:42:13.818 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:42:13.819 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:14.141 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:42:14.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:42:14.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:14.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:14.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:14.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:14.157 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:14.155+0000 7fa60e82d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:14.158 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:14.156+0000 7fa60e82d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:14.159 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:14.157+0000 7fa60e82d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:14.311 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:14.310+0000 7fa60e82d780 -1 Falling back to public interface 2026-02-25T16:42:14.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:42:14.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:14.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:42:14.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:14.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:14.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:14.367 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:14.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:14.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:14.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:42:14.596 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:14.660 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:14.658+0000 7fa60e82d780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:42:15.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:15.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:15.598 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:42:15.825 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2193013717,v1:127.0.0.1:6819/2193013717] [v2:127.0.0.1:6820/2193013717,v1:127.0.0.1:6821/2193013717] exists,up b1251d59-9fcf-4e13-812a-73dfdb7a08aa 2026-02-25T16:42:15.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:15.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:15.830 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:15.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:15.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:15.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:15.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:42:15.832 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:15.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cf7902bf-6e4e-4fab-9e1b-33f69f713880 2026-02-25T16:42:15.832 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 cf7902bf-6e4e-4fab-9e1b-33f69f713880 2026-02-25T16:42:15.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 cf7902bf-6e4e-4fab-9e1b-33f69f713880' 2026-02-25T16:42:15.833 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:15.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBnJp9pmbzzMRAANjITQUsrfrvBnprJx4sM/A== 2026-02-25T16:42:15.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBnJp9pmbzzMRAANjITQUsrfrvBnprJx4sM/A=="}' 2026-02-25T16:42:15.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cf7902bf-6e4e-4fab-9e1b-33f69f713880 -i td/test-erasure-eio/3/new.json 2026-02-25T16:42:16.068 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:42:16.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:42:16.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnJp9pmbzzMRAANjITQUsrfrvBnprJx4sM/A== --osd-uuid cf7902bf-6e4e-4fab-9e1b-33f69f713880 2026-02-25T16:42:16.094 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.092+0000 7fae36051780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:16.095 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.094+0000 7fae36051780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:16.096 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.095+0000 7fae36051780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:16.097 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.095+0000 7fae36051780 -1 bdev(0x3366c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:16.097 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.095+0000 7fae36051780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:42:16.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:42:16.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:16.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:42:16.423 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:42:16.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:16.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:42:16.742 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:42:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:16.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:16.759 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.757+0000 7fb5e5c5b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:16.760 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.759+0000 7fb5e5c5b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:16.761 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.760+0000 7fb5e5c5b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:16.905 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:16.903+0000 7fb5e5c5b780 -1 Falling back to public interface 2026-02-25T16:42:16.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:42:16.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:16.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:42:16.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:16.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:16.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:16.968 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:16.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:16.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:16.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:42:17.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:17.228 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:17.227+0000 7fb5e5c5b780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:42:18.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:18.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:18.193 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:18.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:18.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:18.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:42:18.426 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2464306920,v1:127.0.0.1:6827/2464306920] [v2:127.0.0.1:6828/2464306920,v1:127.0.0.1:6829/2464306920] exists,up cf7902bf-6e4e-4fab-9e1b-33f69f713880 2026-02-25T16:42:18.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:18.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:18.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:18.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:18.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:18.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:18.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T16:42:18.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d41c0eec-88fd-43fe-a9d1-984912500647 2026-02-25T16:42:18.434 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 d41c0eec-88fd-43fe-a9d1-984912500647 2026-02-25T16:42:18.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 d41c0eec-88fd-43fe-a9d1-984912500647' 2026-02-25T16:42:18.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:18.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBqJp9plxw0GhAAV8UmXcGIBH1NdNztKCnhhA== 2026-02-25T16:42:18.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBqJp9plxw0GhAAV8UmXcGIBH1NdNztKCnhhA=="}' 2026-02-25T16:42:18.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d41c0eec-88fd-43fe-a9d1-984912500647 -i td/test-erasure-eio/4/new.json 2026-02-25T16:42:18.669 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:42:18.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T16:42:18.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqJp9plxw0GhAAV8UmXcGIBH1NdNztKCnhhA== --osd-uuid d41c0eec-88fd-43fe-a9d1-984912500647 2026-02-25T16:42:18.695 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:18.693+0000 7f99a9c0b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:18.696 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:18.695+0000 7f99a9c0b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:18.697 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:18.696+0000 7f99a9c0b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:18.698 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:18.696+0000 7f99a9c0b780 -1 bdev(0x28cec00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:18.698 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:18.696+0000 7f99a9c0b780 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T16:42:19.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T16:42:19.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:19.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:42:19.044 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:42:19.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:19.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:42:19.365 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:42:19.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:19.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:19.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:19.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:19.381 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:19.379+0000 7f5f8ee32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:19.382 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:19.381+0000 7f5f8ee32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:19.383 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:19.382+0000 7f5f8ee32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:19.519 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:19.517+0000 7f5f8ee32780 -1 Falling back to public interface 2026-02-25T16:42:19.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:42:19.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:19.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:42:19.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:19.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:19.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:19.591 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:19.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:19.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:19.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:42:19.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:19.881 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:19.879+0000 7f5f8ee32780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:42:20.823 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:20.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:20.824 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:20.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:20.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:20.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:42:21.052 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 34 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3480698425,v1:127.0.0.1:6835/3480698425] [v2:127.0.0.1:6836/3480698425,v1:127.0.0.1:6837/3480698425] exists,up d41c0eec-88fd-43fe-a9d1-984912500647 2026-02-25T16:42:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T16:42:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:21.058 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:21.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:21.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T16:42:21.060 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:21.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fac810ef-c77d-436a-b437-46be1da8855d 2026-02-25T16:42:21.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 fac810ef-c77d-436a-b437-46be1da8855d' 2026-02-25T16:42:21.060 INFO:tasks.workunit.client.0.trial129.stdout:add osd5 fac810ef-c77d-436a-b437-46be1da8855d 2026-02-25T16:42:21.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:21.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBtJp9pJ2ThAxAA93fNSlu9y+6wr9F2CbAY8w== 2026-02-25T16:42:21.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBtJp9pJ2ThAxAA93fNSlu9y+6wr9F2CbAY8w=="}' 2026-02-25T16:42:21.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fac810ef-c77d-436a-b437-46be1da8855d -i td/test-erasure-eio/5/new.json 2026-02-25T16:42:21.295 INFO:tasks.workunit.client.0.trial129.stdout:5 2026-02-25T16:42:21.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T16:42:21.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtJp9pJ2ThAxAA93fNSlu9y+6wr9F2CbAY8w== --osd-uuid fac810ef-c77d-436a-b437-46be1da8855d 2026-02-25T16:42:21.321 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.319+0000 7f3417652780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:21.322 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.321+0000 7f3417652780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:21.323 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.322+0000 7f3417652780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:21.324 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.322+0000 7f3417652780 -1 bdev(0x31d0c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:21.324 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.322+0000 7f3417652780 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T16:42:21.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T16:42:21.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:21.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T16:42:21.655 INFO:tasks.workunit.client.0.trial129.stdout:adding osd5 key to auth repository 2026-02-25T16:42:21.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:21.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T16:42:21.970 INFO:tasks.workunit.client.0.trial129.stdout:start osd.5 2026-02-25T16:42:21.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:21.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:21.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:21.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:21.986 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.984+0000 7f7a9963e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:21.987 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.985+0000 7f7a9963e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:21.988 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:21.986+0000 7f7a9963e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:22.112 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:22.110+0000 7f7a9963e780 -1 Falling back to public interface 2026-02-25T16:42:22.197 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T16:42:22.197 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:22.197 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T16:42:22.197 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:22.197 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:22.198 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:22.198 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:22.198 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:22.198 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:22.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:42:22.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:22.522 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:22.521+0000 7f7a9963e780 -1 osd.5 0 log_to_monitors true 2026-02-25T16:42:23.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:23.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:23.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:23.425 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:23.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:23.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:42:23.649 INFO:tasks.workunit.client.0.trial129.stdout:osd.5 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3460617097,v1:127.0.0.1:6843/3460617097] [v2:127.0.0.1:6844/3460617097,v1:127.0.0.1:6845/3460617097] exists,up fac810ef-c77d-436a-b437-46be1da8855d 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:23.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:23.651 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c241ad9b-b2c6-487b-a919-dd23c2534685 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 c241ad9b-b2c6-487b-a919-dd23c2534685' 2026-02-25T16:42:23.652 INFO:tasks.workunit.client.0.trial129.stdout:add osd6 c241ad9b-b2c6-487b-a919-dd23c2534685 2026-02-25T16:42:23.653 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:23.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBvJp9pXO2AJxAAHg2PUSGMh5mK6Lc90Ac/jg== 2026-02-25T16:42:23.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBvJp9pXO2AJxAAHg2PUSGMh5mK6Lc90Ac/jg=="}' 2026-02-25T16:42:23.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c241ad9b-b2c6-487b-a919-dd23c2534685 -i td/test-erasure-eio/6/new.json 2026-02-25T16:42:23.896 INFO:tasks.workunit.client.0.trial129.stdout:6 2026-02-25T16:42:23.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T16:42:23.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvJp9pXO2AJxAAHg2PUSGMh5mK6Lc90Ac/jg== --osd-uuid c241ad9b-b2c6-487b-a919-dd23c2534685 2026-02-25T16:42:23.924 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:23.923+0000 7f700b025780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:23.926 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:23.924+0000 7f700b025780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:23.926 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:23.925+0000 7f700b025780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:23.927 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:23.925+0000 7f700b025780 -1 bdev(0x2b22c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:23.927 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:23.926+0000 7f700b025780 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T16:42:24.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T16:42:24.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:24.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T16:42:24.308 INFO:tasks.workunit.client.0.trial129.stdout:adding osd6 key to auth repository 2026-02-25T16:42:24.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:24.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T16:42:24.625 INFO:tasks.workunit.client.0.trial129.stdout:start osd.6 2026-02-25T16:42:24.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:24.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:24.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:24.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:24.641 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:24.640+0000 7f6ac5444780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:24.643 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:24.641+0000 7f6ac5444780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:24.644 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:24.642+0000 7f6ac5444780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:24.802 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:24.801+0000 7f6ac5444780 -1 Falling back to public interface 2026-02-25T16:42:24.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:24.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:42:25.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:25.117 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:25.115+0000 7f6ac5444780 -1 osd.6 0 log_to_monitors true 2026-02-25T16:42:26.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:26.086 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:26.086 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:26.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:26.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:42:26.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:26.316 INFO:tasks.workunit.client.0.trial129.stdout:osd.6 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/780886525,v1:127.0.0.1:6851/780886525] [v2:127.0.0.1:6852/780886525,v1:127.0.0.1:6853/780886525] exists,up c241ad9b-b2c6-487b-a919-dd23c2534685 2026-02-25T16:42:26.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:26.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:26.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:26.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:42:26.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:42:26.363 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:42:26.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:42:26.369 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:42:09.347+0000 7f93ace2f780 0 load: jerasure load: lrc 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:477: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:479: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T16:42:26.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:42:26.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T16:42:26.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:42:26.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:42:27.024 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:42:27.032 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:42:28.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:42:28.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:42:28.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:42:28.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:42:28.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:42:28.035 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:42:28.035 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:42:28.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:42:28.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:42:28.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:42:28.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:42:28.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:42:28.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:42:28.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:42:28.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:42:28.123 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:28.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:42:28.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803783 2026-02-25T16:42:28.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803783 2026-02-25T16:42:28.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783' 2026-02-25T16:42:28.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:28.409 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:42:28.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607558 2026-02-25T16:42:28.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607558 2026-02-25T16:42:28.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607558' 2026-02-25T16:42:28.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:28.473 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:42:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444037 2026-02-25T16:42:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444037 2026-02-25T16:42:28.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607558 2-73014444037' 2026-02-25T16:42:28.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:28.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:42:28.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182406 2026-02-25T16:42:28.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182406 2026-02-25T16:42:28.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607558 2-73014444037 3-107374182406' 2026-02-25T16:42:28.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:28.602 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:42:28.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888068 2026-02-25T16:42:28.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888068 2026-02-25T16:42:28.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607558 2-73014444037 3-107374182406 4-146028888068' 2026-02-25T16:42:28.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:28.665 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:42:28.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626437 2026-02-25T16:42:28.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626437 2026-02-25T16:42:28.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607558 2-73014444037 3-107374182406 4-146028888068 5-180388626437' 2026-02-25T16:42:28.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:28.728 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:42:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397507 2026-02-25T16:42:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397507 2026-02-25T16:42:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803783 1-51539607558 2-73014444037 3-107374182406 4-146028888068 5-180388626437 6-210453397507' 2026-02-25T16:42:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:28.791 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803783 2026-02-25T16:42:28.791 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:28.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:42:28.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803783 2026-02-25T16:42:28.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:28.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803783 2026-02-25T16:42:28.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803783' 2026-02-25T16:42:28.794 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803783 2026-02-25T16:42:28.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:42:29.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803783 2026-02-25T16:42:29.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:42:30.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:42:30.023 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:42:30.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803783 2026-02-25T16:42:30.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:30.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607558 2026-02-25T16:42:30.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:30.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:42:30.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607558 2026-02-25T16:42:30.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:30.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607558 2026-02-25T16:42:30.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607558' 2026-02-25T16:42:30.252 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607558 2026-02-25T16:42:30.253 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:42:30.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607558 -lt 51539607558 2026-02-25T16:42:30.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:30.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444037 2026-02-25T16:42:30.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:30.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:42:30.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444037 2026-02-25T16:42:30.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:30.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444037 2026-02-25T16:42:30.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444037' 2026-02-25T16:42:30.480 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 73014444037 2026-02-25T16:42:30.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:42:30.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444037 -lt 73014444037 2026-02-25T16:42:30.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:30.704 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-107374182406 2026-02-25T16:42:30.704 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:30.705 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:42:30.706 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-107374182406 2026-02-25T16:42:30.706 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:30.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=107374182406 2026-02-25T16:42:30.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 107374182406' 2026-02-25T16:42:30.707 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 107374182406 2026-02-25T16:42:30.707 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:42:30.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182406 -lt 107374182406 2026-02-25T16:42:30.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:30.934 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888068 2026-02-25T16:42:30.934 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:30.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:42:30.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888068 2026-02-25T16:42:30.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:30.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888068 2026-02-25T16:42:30.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888068' 2026-02-25T16:42:30.937 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 146028888068 2026-02-25T16:42:30.937 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:42:31.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888068 -lt 146028888068 2026-02-25T16:42:31.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:31.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-180388626437 2026-02-25T16:42:31.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:31.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:42:31.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-180388626437 2026-02-25T16:42:31.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:31.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=180388626437 2026-02-25T16:42:31.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 180388626437' 2026-02-25T16:42:31.165 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 180388626437 2026-02-25T16:42:31.166 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:42:31.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 180388626437 -lt 180388626437 2026-02-25T16:42:31.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:31.391 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-210453397507 2026-02-25T16:42:31.391 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:31.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:42:31.393 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-210453397507 2026-02-25T16:42:31.393 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:31.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397507 2026-02-25T16:42:31.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 210453397507' 2026-02-25T16:42:31.395 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 210453397507 2026-02-25T16:42:31.395 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:42:31.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397508 -lt 210453397507 2026-02-25T16:42:31.619 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:42:31.619 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:42:31.619 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:31.934 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:42:31.934 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:42:31.934 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:42:31.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:42:31.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:42:31.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:42:31.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:42:31.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:42:32.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:42:32.160 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:42:32.160 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:42:32.160 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:32.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:482: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:42:32.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:42:32.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:42:32.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2026-02-25T16:42:32.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2026-02-25T16:42:32.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:42:32.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:42:32.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2026-02-25T16:42:32.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:42:32.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2026-02-25T16:42:32.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2026-02-25T16:42:32.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:42:32.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2026-02-25T16:42:32.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects: ceph osd out 0 2026-02-25T16:42:32.805 INFO:tasks.workunit.client.0.trial129.stderr:osd.0 is already out. 2026-02-25T16:42:32.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects: wait_for_clean 2026-02-25T16:42:32.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:42:32.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:42:32.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:42:32.815 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:42:32.815 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:42:32.815 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:42:32.815 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:42:32.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:42:32.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:42:32.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:42:32.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:42:32.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:42:32.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:42:32.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:42:32.903 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:42:33.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:33.129 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:42:33.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803786 2026-02-25T16:42:33.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803786 2026-02-25T16:42:33.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786' 2026-02-25T16:42:33.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:33.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:42:33.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607561 2026-02-25T16:42:33.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607561 2026-02-25T16:42:33.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607561' 2026-02-25T16:42:33.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:33.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:42:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444040 2026-02-25T16:42:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444040 2026-02-25T16:42:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607561 2-73014444040' 2026-02-25T16:42:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:33.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:42:33.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182409 2026-02-25T16:42:33.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182409 2026-02-25T16:42:33.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607561 2-73014444040 3-107374182409' 2026-02-25T16:42:33.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:33.385 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:42:33.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888071 2026-02-25T16:42:33.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888071 2026-02-25T16:42:33.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607561 2-73014444040 3-107374182409 4-146028888071' 2026-02-25T16:42:33.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:33.449 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:42:33.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626439 2026-02-25T16:42:33.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626439 2026-02-25T16:42:33.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607561 2-73014444040 3-107374182409 4-146028888071 5-180388626439' 2026-02-25T16:42:33.517 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:33.517 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:42:33.579 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397510 2026-02-25T16:42:33.579 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397510 2026-02-25T16:42:33.579 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803786 1-51539607561 2-73014444040 3-107374182409 4-146028888071 5-180388626439 6-210453397510' 2026-02-25T16:42:33.579 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:33.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803786 2026-02-25T16:42:33.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:33.580 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:42:33.581 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803786 2026-02-25T16:42:33.581 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:33.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803786 2026-02-25T16:42:33.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803786' 2026-02-25T16:42:33.582 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803786 2026-02-25T16:42:33.583 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:42:33.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803786 -lt 25769803786 2026-02-25T16:42:33.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:33.812 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607561 2026-02-25T16:42:33.813 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:33.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:42:33.814 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607561 2026-02-25T16:42:33.814 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:33.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607561 2026-02-25T16:42:33.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607561' 2026-02-25T16:42:33.816 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607561 2026-02-25T16:42:33.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:42:34.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607561 -lt 51539607561 2026-02-25T16:42:34.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:34.041 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444040 2026-02-25T16:42:34.042 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:34.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:42:34.043 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444040 2026-02-25T16:42:34.043 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:34.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444040 2026-02-25T16:42:34.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444040' 2026-02-25T16:42:34.044 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 73014444040 2026-02-25T16:42:34.044 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:42:34.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444040 -lt 73014444040 2026-02-25T16:42:34.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:34.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-107374182409 2026-02-25T16:42:34.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:34.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:42:34.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-107374182409 2026-02-25T16:42:34.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:34.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=107374182409 2026-02-25T16:42:34.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 107374182409' 2026-02-25T16:42:34.271 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 107374182409 2026-02-25T16:42:34.271 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:42:34.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182406 -lt 107374182409 2026-02-25T16:42:34.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:42:35.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:42:35.500 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:42:35.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 107374182409 -lt 107374182409 2026-02-25T16:42:35.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:35.733 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888071 2026-02-25T16:42:35.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:35.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:42:35.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888071 2026-02-25T16:42:35.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:35.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888071 2026-02-25T16:42:35.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888071' 2026-02-25T16:42:35.736 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 146028888071 2026-02-25T16:42:35.737 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:42:35.960 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888071 -lt 146028888071 2026-02-25T16:42:35.960 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:35.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-180388626439 2026-02-25T16:42:35.961 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:35.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:42:35.962 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-180388626439 2026-02-25T16:42:35.962 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:35.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=180388626439 2026-02-25T16:42:35.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 180388626439' 2026-02-25T16:42:35.963 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 180388626439 2026-02-25T16:42:35.963 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:42:36.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 180388626440 -lt 180388626439 2026-02-25T16:42:36.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:36.189 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-210453397510 2026-02-25T16:42:36.189 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:36.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:42:36.190 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-210453397510 2026-02-25T16:42:36.190 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:36.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397510 2026-02-25T16:42:36.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 210453397510' 2026-02-25T16:42:36.192 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 210453397510 2026-02-25T16:42:36.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:42:36.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397510 -lt 210453397510 2026-02-25T16:42:36.416 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:42:36.417 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:42:36.417 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:36.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:42:36.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:42:36.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:42:36.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:42:36.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:42:36.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:42:36.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:42:36.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:42:36.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:42:36.954 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:42:36.954 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:42:36.954 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:42:37.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2026-02-25T16:42:37.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:42:37.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:42:37.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2026-02-25T16:42:37.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:42:37.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:42:37.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2026-02-25T16:42:37.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:42:37.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:42:37.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2026-02-25T16:42:37.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:42:37.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:42:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2026-02-25T16:42:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:42:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:42:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2026-02-25T16:42:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:42:37.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:42:37.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2026-02-25T16:42:37.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:42:37.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:42:37.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:495: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:42:37.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:42:37.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:42:37.572 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:42:37.580 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:42:37.859 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:42:37.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:42:37.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:42:37.977 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:42:37.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:42:37.978 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:42:37.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:42:37.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:42:37.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:42:37.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:37.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:42:37.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:42:37.981 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:37.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:42:37.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:42:37.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:42:37.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:42:37.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:37.998 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:37.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:42:37.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:42:38.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:42:38.001 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:42:38.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:42:38.002 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:42:38.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:42:38.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:42:38.004 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:42:38.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:38.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:42:38.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:42:38.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:38.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:42:38.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:42:38.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:42:38.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:42:38.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:38.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:38.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:42:38.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:42:38.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:42:38.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:42:38.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:42:38.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:38.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:38.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:42:38.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:42:38.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:42:38.030 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:42:38.030 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:38.031 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:38.031 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:38.031 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:38.031 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:38.031 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:38.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:42:38.054 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:42:38.055 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:42:38.055 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:42:38.055 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:42:38.055 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:42:38.055 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:42:38.056 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:42:38.056 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:42:38.056 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:42:38.057 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:38.057 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:38.057 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:38.057 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:42:38.057 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:42:38.057 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:42:38.100 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:42:38.100 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:42:38.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:42:38.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:42:38.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:42:38.102 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:42:38.102 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:42:38.102 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:42:38.102 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:38.102 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:38.102 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:38.103 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:42:38.103 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:42:38.103 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:42:38.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:42:38.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:42:38.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:42:38.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:42:38.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:42:38.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:42:38.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:42:38.243 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:42:38.243 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:38.243 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:38.243 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:38.243 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:38.243 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:38.243 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:38.244 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:42:38.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:42:38.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:42:38.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:42:38.357 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:42:38.363 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:42:39.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:42:39.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:42:39.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:42:39.365 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:39.365 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:39.365 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:39.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:42:39.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:42:39.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:42:39.404 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:42:39.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:42:39.410 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:42:38.045+0000 7f4219e1fc00 0 load: jerasure load: lrc 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:501: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:505: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:42:39.411 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T16:42:39.412 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:42:39.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:39.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2c42121e-abef-4d7c-b245-c06926f51e97 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2c42121e-abef-4d7c-b245-c06926f51e97' 2026-02-25T16:42:39.417 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 2c42121e-abef-4d7c-b245-c06926f51e97 2026-02-25T16:42:39.418 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:39.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB/Jp9p7dJtGRAA5drUTrOvxZj2Pe2YaHprNg== 2026-02-25T16:42:39.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB/Jp9p7dJtGRAA5drUTrOvxZj2Pe2YaHprNg=="}' 2026-02-25T16:42:39.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2c42121e-abef-4d7c-b245-c06926f51e97 -i td/test-erasure-eio/0/new.json 2026-02-25T16:42:39.533 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:39.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:42:39.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/Jp9p7dJtGRAA5drUTrOvxZj2Pe2YaHprNg== --osd-uuid 2c42121e-abef-4d7c-b245-c06926f51e97 2026-02-25T16:42:39.560 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:39.559+0000 7f20e000c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:39.562 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:39.561+0000 7f20e000c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:39.563 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:39.562+0000 7f20e000c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:39.564 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:39.562+0000 7f20e000c780 -1 bdev(0x36e2c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:39.564 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:39.562+0000 7f20e000c780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:42:39.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:42:39.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:39.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:42:39.974 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:42:39.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:40.077 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:42:40.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:42:40.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:40.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:40.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:40.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:40.095 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:40.093+0000 7efe323fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:40.096 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:40.095+0000 7efe323fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:40.097 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:40.096+0000 7efe323fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:40.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:42:40.284 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:40.314 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:40.313+0000 7efe323fb780 -1 Falling back to public interface 2026-02-25T16:42:40.576 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:40.574+0000 7efe323fb780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:42:41.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:41.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:41.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:41.286 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:41.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:41.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2416095394,v1:127.0.0.1:6803/2416095394] [v2:127.0.0.1:6804/2416095394,v1:127.0.0.1:6805/2416095394] exists,up 2c42121e-abef-4d7c-b245-c06926f51e97 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:42:41.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:41.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:42:41.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:41.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:41.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:41.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:41.520 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:41.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e6ff57d3-0fe9-4ab5-b315-86d5b1b673d8 2026-02-25T16:42:41.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e6ff57d3-0fe9-4ab5-b315-86d5b1b673d8' 2026-02-25T16:42:41.525 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 e6ff57d3-0fe9-4ab5-b315-86d5b1b673d8 2026-02-25T16:42:41.525 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:41.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBJp9pmRmfHxAAKa4jQf3AZKtEO3vcJe7BJQ== 2026-02-25T16:42:41.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBJp9pmRmfHxAAKa4jQf3AZKtEO3vcJe7BJQ=="}' 2026-02-25T16:42:41.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e6ff57d3-0fe9-4ab5-b315-86d5b1b673d8 -i td/test-erasure-eio/1/new.json 2026-02-25T16:42:41.765 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:41.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:42:41.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBJp9pmRmfHxAAKa4jQf3AZKtEO3vcJe7BJQ== --osd-uuid e6ff57d3-0fe9-4ab5-b315-86d5b1b673d8 2026-02-25T16:42:41.790 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:41.789+0000 7fed53e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:41.791 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:41.790+0000 7fed53e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:41.792 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:41.791+0000 7fed53e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:41.793 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:41.791+0000 7fed53e2c780 -1 bdev(0x29c0c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:41.793 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:41.791+0000 7fed53e2c780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:42:42.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:42:42.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:42.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:42:42.133 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:42:42.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:42.452 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:42:42.453 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:42:42.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:42.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:42.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:42.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:42.468 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:42.466+0000 7f82efdfd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:42.469 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:42.468+0000 7f82efdfd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:42.470 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:42.469+0000 7f82efdfd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:42.656 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:42.655+0000 7f82efdfd780 -1 Falling back to public interface 2026-02-25T16:42:42.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:42:42.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:42.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:42:42.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:42.678 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:42.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:42.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:42.679 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:42.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:42.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:42:42.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:43.005 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:43.003+0000 7f82efdfd780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:42:43.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:43.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:43.908 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:43.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:43.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:43.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:42:44.132 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3062246753,v1:127.0.0.1:6811/3062246753] [v2:127.0.0.1:6812/3062246753,v1:127.0.0.1:6813/3062246753] exists,up e6ff57d3-0fe9-4ab5-b315-86d5b1b673d8 2026-02-25T16:42:44.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:44.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:44.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:44.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:44.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:42:44.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:44.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:44.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f8aed678-5b5d-41ac-ba3c-060f22364a16 2026-02-25T16:42:44.140 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 f8aed678-5b5d-41ac-ba3c-060f22364a16 2026-02-25T16:42:44.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f8aed678-5b5d-41ac-ba3c-060f22364a16' 2026-02-25T16:42:44.140 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:44.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCEJp9peEyzCBAABcssGEQB1DYJacAlP0qM8g== 2026-02-25T16:42:44.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCEJp9peEyzCBAABcssGEQB1DYJacAlP0qM8g=="}' 2026-02-25T16:42:44.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f8aed678-5b5d-41ac-ba3c-060f22364a16 -i td/test-erasure-eio/2/new.json 2026-02-25T16:42:44.378 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:42:44.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:42:44.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEJp9peEyzCBAABcssGEQB1DYJacAlP0qM8g== --osd-uuid f8aed678-5b5d-41ac-ba3c-060f22364a16 2026-02-25T16:42:44.404 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:44.402+0000 7f568320f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:44.405 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:44.403+0000 7f568320f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:44.406 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:44.404+0000 7f568320f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:44.406 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:44.405+0000 7f568320f780 -1 bdev(0x31bcc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:44.406 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:44.405+0000 7f568320f780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:42:44.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:42:44.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:44.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:42:44.753 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:42:44.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:45.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:42:45.071 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:42:45.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:45.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:45.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:45.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:45.086 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:45.084+0000 7fb13d647780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:45.088 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:45.086+0000 7fb13d647780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:45.089 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:45.087+0000 7fb13d647780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:45.237 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:45.236+0000 7fb13d647780 -1 Falling back to public interface 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:45.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:42:45.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:45.621 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:45.619+0000 7fb13d647780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:42:46.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:46.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:46.532 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:46.532 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:46.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:46.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:42:46.760 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4186940153,v1:127.0.0.1:6819/4186940153] [v2:127.0.0.1:6820/4186940153,v1:127.0.0.1:6821/4186940153] exists,up f8aed678-5b5d-41ac-ba3c-060f22364a16 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:42:46.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:46.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:42:46.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:46.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=011c6258-25fd-4ee2-8efc-43b182fb8a30 2026-02-25T16:42:46.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 011c6258-25fd-4ee2-8efc-43b182fb8a30' 2026-02-25T16:42:46.764 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 011c6258-25fd-4ee2-8efc-43b182fb8a30 2026-02-25T16:42:46.764 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:46.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGJp9pHZgpLhAAKAkocJwaqzFgrsX/giIL2A== 2026-02-25T16:42:46.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGJp9pHZgpLhAAKAkocJwaqzFgrsX/giIL2A=="}' 2026-02-25T16:42:46.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 011c6258-25fd-4ee2-8efc-43b182fb8a30 -i td/test-erasure-eio/3/new.json 2026-02-25T16:42:47.009 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:42:47.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:42:47.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGJp9pHZgpLhAAKAkocJwaqzFgrsX/giIL2A== --osd-uuid 011c6258-25fd-4ee2-8efc-43b182fb8a30 2026-02-25T16:42:47.035 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.034+0000 7f68cfe1e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:47.037 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.035+0000 7f68cfe1e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:47.038 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.036+0000 7f68cfe1e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:47.038 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.036+0000 7f68cfe1e780 -1 bdev(0x3be8c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:47.038 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.036+0000 7f68cfe1e780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:42:47.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:42:47.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:47.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:42:47.437 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:42:47.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:47.765 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:42:47.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:42:47.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:47.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:47.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:47.765 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:47.781 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.780+0000 7fc4b1458780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:47.783 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.781+0000 7fc4b1458780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:47.784 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.782+0000 7fc4b1458780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:47.951 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:47.950+0000 7fc4b1458780 -1 Falling back to public interface 2026-02-25T16:42:47.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:42:47.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:47.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:42:47.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:47.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:47.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:47.996 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:47.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:47.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:47.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:42:48.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:48.279 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:48.277+0000 7fc4b1458780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:42:49.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:49.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:49.224 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:49.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:49.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:49.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:42:49.453 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2793559421,v1:127.0.0.1:6827/2793559421] [v2:127.0.0.1:6828/2793559421,v1:127.0.0.1:6829/2793559421] exists,up 011c6258-25fd-4ee2-8efc-43b182fb8a30 2026-02-25T16:42:49.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:49.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:49.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:49.455 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:49.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=20486468-3911-4207-9098-3a24a5d1311e 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 20486468-3911-4207-9098-3a24a5d1311e' 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 20486468-3911-4207-9098-3a24a5d1311e 2026-02-25T16:42:49.457 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:49.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCJJp9ppVfhGxAAJGVuP+PZ2w7+/UY0k+wx4A== 2026-02-25T16:42:49.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCJJp9ppVfhGxAAJGVuP+PZ2w7+/UY0k+wx4A=="}' 2026-02-25T16:42:49.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 20486468-3911-4207-9098-3a24a5d1311e -i td/test-erasure-eio/4/new.json 2026-02-25T16:42:49.700 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:42:49.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T16:42:49.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJJp9ppVfhGxAAJGVuP+PZ2w7+/UY0k+wx4A== --osd-uuid 20486468-3911-4207-9098-3a24a5d1311e 2026-02-25T16:42:49.726 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:49.724+0000 7f451ba2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:49.727 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:49.725+0000 7f451ba2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:49.728 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:49.726+0000 7f451ba2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:49.729 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:49.727+0000 7f451ba2c780 -1 bdev(0x298ec00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:49.729 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:49.727+0000 7f451ba2c780 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T16:42:50.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T16:42:50.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:50.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:42:50.058 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:42:50.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:50.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:42:50.376 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:42:50.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:50.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:50.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:50.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:50.393 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:50.391+0000 7f73cee2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:50.394 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:50.392+0000 7f73cee2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:50.395 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:50.393+0000 7f73cee2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:50.537 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:50.535+0000 7f73cee2d780 -1 Falling back to public interface 2026-02-25T16:42:50.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:42:50.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:50.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:42:50.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:50.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:50.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:50.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:50.608 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:50.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:50.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:42:50.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:50.839 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:50.837+0000 7f73cee2d780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:42:51.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:51.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:51.840 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:51.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:51.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:51.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:42:52.068 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 35 up_thru 36 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3512617380,v1:127.0.0.1:6835/3512617380] [v2:127.0.0.1:6836/3512617380,v1:127.0.0.1:6837/3512617380] exists,up 20486468-3911-4207-9098-3a24a5d1311e 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T16:42:52.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:52.073 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:52.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:52.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4606d6af-a000-4450-b2a6-a4744160069e 2026-02-25T16:42:52.076 INFO:tasks.workunit.client.0.trial129.stdout:add osd5 4606d6af-a000-4450-b2a6-a4744160069e 2026-02-25T16:42:52.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 4606d6af-a000-4450-b2a6-a4744160069e' 2026-02-25T16:42:52.076 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:52.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMJp9pGSPhBBAAy/skxZVU4Yl4ripokjYcRg== 2026-02-25T16:42:52.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMJp9pGSPhBBAAy/skxZVU4Yl4ripokjYcRg=="}' 2026-02-25T16:42:52.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4606d6af-a000-4450-b2a6-a4744160069e -i td/test-erasure-eio/5/new.json 2026-02-25T16:42:52.315 INFO:tasks.workunit.client.0.trial129.stdout:5 2026-02-25T16:42:52.323 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T16:42:52.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMJp9pGSPhBBAAy/skxZVU4Yl4ripokjYcRg== --osd-uuid 4606d6af-a000-4450-b2a6-a4744160069e 2026-02-25T16:42:52.341 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:52.339+0000 7f4071636780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:52.342 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:52.341+0000 7f4071636780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:52.343 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:52.342+0000 7f4071636780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:52.343 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:52.342+0000 7f4071636780 -1 bdev(0x295ec00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:52.343 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:52.342+0000 7f4071636780 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T16:42:52.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T16:42:52.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:52.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T16:42:52.695 INFO:tasks.workunit.client.0.trial129.stdout:adding osd5 key to auth repository 2026-02-25T16:42:52.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:53.014 INFO:tasks.workunit.client.0.trial129.stdout:start osd.5 2026-02-25T16:42:53.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T16:42:53.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:53.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:53.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:53.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:53.031 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:53.029+0000 7f7b0a209780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:53.032 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:53.031+0000 7f7b0a209780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:53.033 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:53.032+0000 7f7b0a209780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:53.198 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:53.196+0000 7f7b0a209780 -1 Falling back to public interface 2026-02-25T16:42:53.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T16:42:53.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:53.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T16:42:53.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:53.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:53.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:53.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:53.248 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:53.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:53.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:42:53.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:53.529 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:53.527+0000 7f7b0a209780 -1 osd.5 0 log_to_monitors true 2026-02-25T16:42:54.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:54.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:54.477 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:54.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:54.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:54.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:42:54.708 INFO:tasks.workunit.client.0.trial129.stdout:osd.5 up in weight 1 up_from 43 up_thru 44 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1355180684,v1:127.0.0.1:6843/1355180684] [v2:127.0.0.1:6844/1355180684,v1:127.0.0.1:6845/1355180684] exists,up 4606d6af-a000-4450-b2a6-a4744160069e 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:42:54.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:54.710 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T16:42:54.711 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:42:54.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e433c736-6e0e-40ba-aff9-2cf200c889bd 2026-02-25T16:42:54.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 e433c736-6e0e-40ba-aff9-2cf200c889bd' 2026-02-25T16:42:54.712 INFO:tasks.workunit.client.0.trial129.stdout:add osd6 e433c736-6e0e-40ba-aff9-2cf200c889bd 2026-02-25T16:42:54.712 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:42:54.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCOJp9pBn8GKxAAKUALPd7U8x3dKgNoavKARA== 2026-02-25T16:42:54.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCOJp9pBn8GKxAAKUALPd7U8x3dKgNoavKARA=="}' 2026-02-25T16:42:54.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e433c736-6e0e-40ba-aff9-2cf200c889bd -i td/test-erasure-eio/6/new.json 2026-02-25T16:42:54.958 INFO:tasks.workunit.client.0.trial129.stdout:6 2026-02-25T16:42:54.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T16:42:54.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOJp9pBn8GKxAAKUALPd7U8x3dKgNoavKARA== --osd-uuid e433c736-6e0e-40ba-aff9-2cf200c889bd 2026-02-25T16:42:54.985 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:54.983+0000 7fe7eb40e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:54.986 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:54.984+0000 7fe7eb40e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:54.987 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:54.985+0000 7fe7eb40e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:54.987 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:54.986+0000 7fe7eb40e780 -1 bdev(0x2612c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T16:42:54.988 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:54.986+0000 7fe7eb40e780 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T16:42:55.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T16:42:55.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:42:55.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T16:42:55.347 INFO:tasks.workunit.client.0.trial129.stdout:adding osd6 key to auth repository 2026-02-25T16:42:55.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:42:55.665 INFO:tasks.workunit.client.0.trial129.stdout:start osd.6 2026-02-25T16:42:55.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T16:42:55.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:42:55.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:42:55.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:42:55.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:42:55.681 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:55.680+0000 7f02de814780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:55.683 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:55.681+0000 7f02de814780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:55.683 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:55.682+0000 7f02de814780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:42:55.818 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:55.816+0000 7f02de814780 -1 Falling back to public interface 2026-02-25T16:42:55.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T16:42:55.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:42:55.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T16:42:55.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:42:55.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:42:55.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:55.895 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:42:55.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:42:55.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:55.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:42:56.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:42:56.219 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:42:56.217+0000 7f02de814780 -1 osd.6 0 log_to_monitors true 2026-02-25T16:42:57.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:42:57.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:42:57.121 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:42:57.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:42:57.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:42:57.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:42:57.352 INFO:tasks.workunit.client.0.trial129.stdout:osd.6 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2884207551,v1:127.0.0.1:6851/2884207551] [v2:127.0.0.1:6852/2884207551,v1:127.0.0.1:6853/2884207551] exists,up e433c736-6e0e-40ba-aff9-2cf200c889bd 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:42:57.353 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:42:57.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:42:57.354 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:42:57.354 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:42:57.398 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:42:57.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:42:57.404 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:42:40.315+0000 7efe323fb780 0 load: jerasure load: lrc 2026-02-25T16:42:57.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:508: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:509: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:42:57.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T16:42:57.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:42:57.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:42:58.062 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:42:58.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:42:59.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:42:59.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:42:59.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:42:59.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:42:59.072 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:42:59.072 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:42:59.072 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:42:59.072 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:42:59.072 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:42:59.072 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:42:59.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:42:59.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:42:59.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:42:59.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:42:59.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:42:59.162 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:42:59.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:42:59.384 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:59.385 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:42:59.447 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803782 2026-02-25T16:42:59.447 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803782 2026-02-25T16:42:59.447 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782' 2026-02-25T16:42:59.447 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:59.447 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:42:59.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-25T16:42:59.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-25T16:42:59.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854' 2026-02-25T16:42:59.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:59.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:42:59.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378629 2026-02-25T16:42:59.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378629 2026-02-25T16:42:59.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378629' 2026-02-25T16:42:59.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:59.574 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:42:59.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149701 2026-02-25T16:42:59.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149701 2026-02-25T16:42:59.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378629 3-111669149701' 2026-02-25T16:42:59.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:59.639 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:42:59.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855365 2026-02-25T16:42:59.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855365 2026-02-25T16:42:59.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378629 3-111669149701 4-150323855365' 2026-02-25T16:42:59.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:59.703 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:42:59.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-25T16:42:59.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-25T16:42:59.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378629 3-111669149701 4-150323855365 5-184683593732' 2026-02-25T16:42:59.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:42:59.768 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:42:59.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397507 2026-02-25T16:42:59.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397507 2026-02-25T16:42:59.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378629 3-111669149701 4-150323855365 5-184683593732 6-210453397507' 2026-02-25T16:42:59.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:42:59.831 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803782 2026-02-25T16:42:59.832 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:42:59.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:42:59.833 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803782 2026-02-25T16:42:59.833 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:42:59.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803782 2026-02-25T16:42:59.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803782' 2026-02-25T16:42:59.834 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803782 2026-02-25T16:42:59.834 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:43:00.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803780 -lt 25769803782 2026-02-25T16:43:00.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:43:01.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:43:01.059 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:43:01.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803782 2026-02-25T16:43:01.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:01.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574854 2026-02-25T16:43:01.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:01.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:43:01.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574854 2026-02-25T16:43:01.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:01.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574854 2026-02-25T16:43:01.289 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574854 2026-02-25T16:43:01.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574854' 2026-02-25T16:43:01.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:43:01.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574854 -lt 55834574854 2026-02-25T16:43:01.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:01.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378629 2026-02-25T16:43:01.513 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:01.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:43:01.513 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378629 2026-02-25T16:43:01.513 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:01.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378629 2026-02-25T16:43:01.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378629' 2026-02-25T16:43:01.515 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378629 2026-02-25T16:43:01.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:43:01.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378629 -lt 81604378629 2026-02-25T16:43:01.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:01.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149701 2026-02-25T16:43:01.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:01.743 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:43:01.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149701 2026-02-25T16:43:01.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:01.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149701 2026-02-25T16:43:01.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149701' 2026-02-25T16:43:01.745 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 111669149701 2026-02-25T16:43:01.745 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:43:01.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149701 -lt 111669149701 2026-02-25T16:43:01.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:01.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855365 2026-02-25T16:43:01.969 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:01.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:43:01.970 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855365 2026-02-25T16:43:01.970 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:01.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855365 2026-02-25T16:43:01.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855365' 2026-02-25T16:43:01.972 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855365 2026-02-25T16:43:01.972 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:43:02.201 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855365 -lt 150323855365 2026-02-25T16:43:02.201 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:02.201 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-184683593732 2026-02-25T16:43:02.201 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:02.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:43:02.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-184683593732 2026-02-25T16:43:02.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:02.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593732 2026-02-25T16:43:02.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 184683593732' 2026-02-25T16:43:02.205 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 184683593732 2026-02-25T16:43:02.205 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:43:02.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-25T16:43:02.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:02.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-210453397507 2026-02-25T16:43:02.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:02.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:43:02.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-210453397507 2026-02-25T16:43:02.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:02.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397507 2026-02-25T16:43:02.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 210453397507' 2026-02-25T16:43:02.436 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 210453397507 2026-02-25T16:43:02.437 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:43:02.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397507 -lt 210453397507 2026-02-25T16:43:02.660 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:43:02.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:02.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:43:02.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:43:03.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:43:03.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:43:03.195 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:03.195 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:03.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:43:03.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:43:03.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:43:03.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2026-02-25T16:43:03.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:43:03.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:43:03.509 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:43:03.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:43:03.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:43:03.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:03.544 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2026-02-25T16:43:03.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:43:03.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:43:03.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2026-02-25T16:43:03.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:43:03.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:43:03.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:43:03.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:03.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:43:03.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:03.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:516: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-25T16:43:03.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:43:03.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:43:03.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:43:03.561 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:43:03.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:43:03.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T16:43:03.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:43:03.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T16:43:03.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:43:03.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T16:43:03.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:43:03.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:43:03.790 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:43:03.791 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:43:03.836 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:43:03.836 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:43:03.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:43:03.837 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T16:43:03.837 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:43:03.837 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:43:03.837 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:03.837 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:03.837 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:03.838 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:43:03.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:43:03.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-25T16:43:03.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2026-02-25T16:43:04.165 INFO:tasks.workunit.client.0.trial129.stderr:osd.0 is already out. 2026-02-25T16:43:04.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2026-02-25T16:43:04.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:43:04.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:43:04.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:43:04.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:43:04.174 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:43:04.174 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:43:04.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:43:04.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:43:04.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:43:04.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:43:04.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:43:04.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:43:04.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:43:04.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:43:04.261 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:43:04.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:43:04.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:04.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:43:04.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T16:43:04.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T16:43:04.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T16:43:04.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:04.556 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:43:04.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-25T16:43:04.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-25T16:43:04.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857' 2026-02-25T16:43:04.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:04.623 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:43:04.683 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378632 2026-02-25T16:43:04.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378632 2026-02-25T16:43:04.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632' 2026-02-25T16:43:04.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:04.684 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:43:04.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149704 2026-02-25T16:43:04.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149704 2026-02-25T16:43:04.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149704' 2026-02-25T16:43:04.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:04.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:43:04.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855368 2026-02-25T16:43:04.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855368 2026-02-25T16:43:04.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149704 4-150323855368' 2026-02-25T16:43:04.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:04.811 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:43:04.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593735 2026-02-25T16:43:04.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593735 2026-02-25T16:43:04.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149704 4-150323855368 5-184683593735' 2026-02-25T16:43:04.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:04.874 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:43:04.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397510 2026-02-25T16:43:04.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397510 2026-02-25T16:43:04.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378632 3-111669149704 4-150323855368 5-184683593735 6-210453397510' 2026-02-25T16:43:04.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:04.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T16:43:04.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:04.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:43:04.940 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T16:43:04.940 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:04.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T16:43:04.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T16:43:04.942 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803785 2026-02-25T16:43:04.942 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:43:05.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 0 -lt 25769803785 2026-02-25T16:43:05.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:43:06.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:43:06.169 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:43:06.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T16:43:06.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:06.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574857 2026-02-25T16:43:06.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:06.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:43:06.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574857 2026-02-25T16:43:06.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:06.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574857 2026-02-25T16:43:06.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574857' 2026-02-25T16:43:06.402 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574857 2026-02-25T16:43:06.402 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:43:06.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574857 -lt 55834574857 2026-02-25T16:43:06.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:06.629 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378632 2026-02-25T16:43:06.629 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:06.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:43:06.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378632 2026-02-25T16:43:06.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:06.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378632 2026-02-25T16:43:06.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378632' 2026-02-25T16:43:06.633 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378632 2026-02-25T16:43:06.633 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:43:06.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378632 -lt 81604378632 2026-02-25T16:43:06.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:06.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149704 2026-02-25T16:43:06.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:06.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:43:06.862 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149704 2026-02-25T16:43:06.862 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:06.864 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149704 2026-02-25T16:43:06.864 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149704' 2026-02-25T16:43:06.865 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 111669149704 2026-02-25T16:43:06.865 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:43:07.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149704 -lt 111669149704 2026-02-25T16:43:07.086 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:07.086 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855368 2026-02-25T16:43:07.086 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:07.087 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:43:07.087 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855368 2026-02-25T16:43:07.088 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:07.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855368 2026-02-25T16:43:07.089 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855368 2026-02-25T16:43:07.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855368' 2026-02-25T16:43:07.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:43:07.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855368 -lt 150323855368 2026-02-25T16:43:07.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:07.316 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-184683593735 2026-02-25T16:43:07.316 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:07.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:43:07.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-184683593735 2026-02-25T16:43:07.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:07.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593735 2026-02-25T16:43:07.319 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 184683593735 2026-02-25T16:43:07.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 184683593735' 2026-02-25T16:43:07.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:43:07.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593735 -lt 184683593735 2026-02-25T16:43:07.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:07.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-210453397510 2026-02-25T16:43:07.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:07.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:43:07.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-210453397510 2026-02-25T16:43:07.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397510 2026-02-25T16:43:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 210453397510' 2026-02-25T16:43:07.549 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 210453397510 2026-02-25T16:43:07.549 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:43:07.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397510 -lt 210453397510 2026-02-25T16:43:07.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:43:07.777 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:07.778 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:08.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:43:08.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:43:08.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:43:08.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:43:08.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:43:08.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:43:08.091 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:43:08.091 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:43:08.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:43:08.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:43:08.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:08.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:08.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:43:08.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2026-02-25T16:43:08.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:43:08.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:43:08.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2026-02-25T16:43:08.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:43:08.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:43:08.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2026-02-25T16:43:08.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:43:08.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:43:08.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2026-02-25T16:43:08.668 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:43:08.669 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:43:08.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:524: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2026-02-25T16:43:08.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:43:08.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:43:08.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2026-02-25T16:43:08.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:43:08.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:43:08.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2026-02-25T16:43:08.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:43:08.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:43:08.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:526: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:43:08.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:43:08.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:43:08.961 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:43:08.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:43:09.244 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:43:09.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:43:09.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:43:09.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:43:09.365 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:43:09.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:43:09.366 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:43:09.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:43:09.368 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:43:09.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:43:09.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:43:09.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:43:09.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:43:09.369 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:43:09.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:43:09.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:43:09.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:43:09.385 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:43:09.386 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:09.386 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:09.386 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:43:09.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:43:09.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:43:09.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:43:09.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:43:09.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:43:09.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:43:09.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:43:09.390 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:43:09.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:43:09.391 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:43:09.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:43:09.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:43:09.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:43:09.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:43:09.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:43:09.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:43:09.393 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:43:09.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:43:09.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:43:09.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:43:09.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:43:09.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:09.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:09.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:43:09.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:43:09.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:43:09.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:43:09.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:43:09.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:09.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:09.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:43:09.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:43:09.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:43:09.402 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:43:09.419 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:43:09.419 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:09.419 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:09.420 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:09.420 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:09.420 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:09.420 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:09.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:43:09.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:43:09.445 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:43:09.446 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:09.446 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:09.446 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:09.446 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:43:09.446 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:43:09.446 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:43:09.489 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:43:09.489 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:43:09.489 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:43:09.489 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:43:09.489 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:43:09.489 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:43:09.489 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:43:09.490 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:43:09.490 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:09.490 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:09.490 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:09.490 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:43:09.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:43:09.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:43:09.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:43:09.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:43:09.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:43:09.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:43:09.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:43:09.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:43:09.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:09.628 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:43:09.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:43:09.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:43:09.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:43:09.738 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:43:09.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:43:10.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:43:10.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:43:10.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:43:10.747 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:10.748 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:10.748 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:10.748 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:43:10.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:43:10.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:43:10.785 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:43:10.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:43:10.791 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:43:09.434+0000 7f320f234c00 0 load: jerasure load: lrc 2026-02-25T16:43:10.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2026-02-25T16:43:10.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2026-02-25T16:43:10.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: local objname=myobject 2026-02-25T16:43:10.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: local lastobj=100 2026-02-25T16:43:10.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:633: TEST_ec_recovery_unfound: local testobj=obj75 2026-02-25T16:43:10.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:635: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:43:10.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-25T16:43:10.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:43:10.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:638: TEST_ec_recovery_unfound: setup_osds 5 2026-02-25T16:43:10.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2026-02-25T16:43:10.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:43:10.803 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2026-02-25T16:43:10.803 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2026-02-25T16:43:10.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:43:10.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:43:10.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:43:10.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:43:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:43:10.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fe198e7f-a2be-4ee0-9f96-7786e4ea8901 2026-02-25T16:43:10.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fe198e7f-a2be-4ee0-9f96-7786e4ea8901' 2026-02-25T16:43:10.807 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 fe198e7f-a2be-4ee0-9f96-7786e4ea8901 2026-02-25T16:43:10.807 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:43:10.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCeJp9pqMwwMBAA9xiaoNcU6TerRz+nwfzQ+g== 2026-02-25T16:43:10.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCeJp9pqMwwMBAA9xiaoNcU6TerRz+nwfzQ+g=="}' 2026-02-25T16:43:10.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fe198e7f-a2be-4ee0-9f96-7786e4ea8901 -i td/test-erasure-eio/0/new.json 2026-02-25T16:43:10.914 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:43:10.920 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:43:10.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeJp9pqMwwMBAA9xiaoNcU6TerRz+nwfzQ+g== --osd-uuid fe198e7f-a2be-4ee0-9f96-7786e4ea8901 2026-02-25T16:43:10.942 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:10.940+0000 7f714b83b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:10.943 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:10.942+0000 7f714b83b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:10.944 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:10.942+0000 7f714b83b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:10.945 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:10.943+0000 7f714b83b780 -1 bdev(0x2a86c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:43:10.945 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:10.943+0000 7f714b83b780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:43:11.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:43:11.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:43:11.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:43:11.358 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:43:11.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:43:11.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:43:11.463 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:43:11.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:43:11.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:43:11.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:43:11.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:43:11.483 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:11.481+0000 7f7c7043a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:11.484 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:11.482+0000 7f7c7043a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:11.485 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:11.483+0000 7f7c7043a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:11.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:43:11.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:43:11.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:43:11.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:43:11.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:43:11.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:11.566 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:43:11.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:43:11.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:11.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:43:11.663 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:43:11.666 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:11.665+0000 7f7c7043a780 -1 Falling back to public interface 2026-02-25T16:43:11.940 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:11.938+0000 7f7c7043a780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:43:12.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:43:12.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:12.664 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:43:12.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:43:12.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:12.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:43:12.900 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3944396151,v1:127.0.0.1:6803/3944396151] [v2:127.0.0.1:6804/3944396151,v1:127.0.0.1:6805/3944396151] exists,up fe198e7f-a2be-4ee0-9f96-7786e4ea8901 2026-02-25T16:43:12.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:43:12.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:43:12.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:43:12.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:43:12.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:43:12.905 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:12.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=269077f1-87ac-41f9-a645-c51ae4c26574 2026-02-25T16:43:12.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 269077f1-87ac-41f9-a645-c51ae4c26574' 2026-02-25T16:43:12.908 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 269077f1-87ac-41f9-a645-c51ae4c26574 2026-02-25T16:43:12.908 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:43:12.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgJp9p0iaFNhAA7jft/zZfmCyPWA2esqAQMA== 2026-02-25T16:43:12.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgJp9p0iaFNhAA7jft/zZfmCyPWA2esqAQMA=="}' 2026-02-25T16:43:12.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 269077f1-87ac-41f9-a645-c51ae4c26574 -i td/test-erasure-eio/1/new.json 2026-02-25T16:43:13.153 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:43:13.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:43:13.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgJp9p0iaFNhAA7jft/zZfmCyPWA2esqAQMA== --osd-uuid 269077f1-87ac-41f9-a645-c51ae4c26574 2026-02-25T16:43:13.181 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.180+0000 7f0ea3a12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:13.183 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.181+0000 7f0ea3a12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:13.184 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.182+0000 7f0ea3a12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:13.184 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.182+0000 7f0ea3a12780 -1 bdev(0x3980c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:43:13.184 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.182+0000 7f0ea3a12780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:43:13.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:43:13.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:43:13.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:43:13.511 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:43:13.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:43:13.838 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:43:13.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:43:13.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:43:13.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:43:13.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:43:13.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:43:13.857 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.856+0000 7f3e73a2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:13.859 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.857+0000 7f3e73a2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:13.859 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.858+0000 7f3e73a2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:14.001 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:13.999+0000 7f3e73a2e780 -1 Falling back to public interface 2026-02-25T16:43:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:43:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:43:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:43:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:43:14.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:43:14.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:14.069 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:43:14.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:43:14.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:14.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:43:14.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:43:14.308 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:14.306+0000 7f3e73a2e780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:43:15.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:43:15.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:15.305 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:43:15.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:43:15.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:15.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:43:15.533 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/738029365,v1:127.0.0.1:6811/738029365] [v2:127.0.0.1:6812/738029365,v1:127.0.0.1:6813/738029365] exists,up 269077f1-87ac-41f9-a645-c51ae4c26574 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:43:15.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:43:15.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:43:15.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:43:15.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:15.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:43:15.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f1895396-3d26-4c78-abbe-b2398de2a830 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f1895396-3d26-4c78-abbe-b2398de2a830' 2026-02-25T16:43:15.540 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 f1895396-3d26-4c78-abbe-b2398de2a830 2026-02-25T16:43:15.541 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:43:15.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjJp9piFXDIBAAlXnPiZxK8OqXcqR+TecfHw== 2026-02-25T16:43:15.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjJp9piFXDIBAAlXnPiZxK8OqXcqR+TecfHw=="}' 2026-02-25T16:43:15.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f1895396-3d26-4c78-abbe-b2398de2a830 -i td/test-erasure-eio/2/new.json 2026-02-25T16:43:15.783 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:43:15.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:43:15.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjJp9piFXDIBAAlXnPiZxK8OqXcqR+TecfHw== --osd-uuid f1895396-3d26-4c78-abbe-b2398de2a830 2026-02-25T16:43:15.811 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:15.810+0000 7fa5e2c22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:15.812 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:15.811+0000 7fa5e2c22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:15.813 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:15.812+0000 7fa5e2c22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:15.814 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:15.812+0000 7fa5e2c22780 -1 bdev(0x2a04c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:43:15.814 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:15.812+0000 7fa5e2c22780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:43:16.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:43:16.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:43:16.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:43:16.163 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:43:16.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:43:16.481 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:43:16.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:43:16.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:43:16.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:43:16.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:43:16.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:43:16.500 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:16.498+0000 7fce367fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:16.502 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:16.500+0000 7fce367fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:16.503 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:16.501+0000 7fce367fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:16.656 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:16.655+0000 7fce367fd780 -1 Falling back to public interface 2026-02-25T16:43:16.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:43:16.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:43:16.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:43:16.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:43:16.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:43:16.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:16.713 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:43:16.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:43:16.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:16.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:43:16.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:43:16.964 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:16.962+0000 7fce367fd780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:43:17.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:43:17.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:17.943 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:43:17.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:43:17.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:17.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:43:18.172 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/571657359,v1:127.0.0.1:6819/571657359] [v2:127.0.0.1:6820/571657359,v1:127.0.0.1:6821/571657359] exists,up f1895396-3d26-4c78-abbe-b2398de2a830 2026-02-25T16:43:18.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:43:18.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:43:18.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:43:18.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:18.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:43:18.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:43:18.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:43:18.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=76971875-5666-492d-b6fb-fadde1ccd534 2026-02-25T16:43:18.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 76971875-5666-492d-b6fb-fadde1ccd534' 2026-02-25T16:43:18.181 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 76971875-5666-492d-b6fb-fadde1ccd534 2026-02-25T16:43:18.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:43:18.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmJp9poY4zCxAAghRUZ3SIAYDnsAKFGxuX8w== 2026-02-25T16:43:18.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmJp9poY4zCxAAghRUZ3SIAYDnsAKFGxuX8w=="}' 2026-02-25T16:43:18.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 76971875-5666-492d-b6fb-fadde1ccd534 -i td/test-erasure-eio/3/new.json 2026-02-25T16:43:18.424 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:43:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:43:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmJp9poY4zCxAAghRUZ3SIAYDnsAKFGxuX8w== --osd-uuid 76971875-5666-492d-b6fb-fadde1ccd534 2026-02-25T16:43:18.453 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:18.451+0000 7f6d25c18780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:18.454 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:18.453+0000 7f6d25c18780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:18.455 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:18.453+0000 7f6d25c18780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:18.456 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:18.454+0000 7f6d25c18780 -1 bdev(0x3faec00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:43:18.456 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:18.454+0000 7f6d25c18780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:43:18.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:43:18.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:43:18.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:43:18.793 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:43:18.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:43:19.112 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:43:19.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:43:19.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:43:19.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:43:19.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:43:19.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:43:19.131 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:19.129+0000 7f02e3244780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:19.133 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:19.131+0000 7f02e3244780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:19.133 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:19.132+0000 7f02e3244780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:19.300 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:19.298+0000 7f02e3244780 -1 Falling back to public interface 2026-02-25T16:43:19.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:43:19.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:43:19.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:43:19.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:43:19.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:43:19.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:19.345 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:43:19.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:43:19.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:19.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:43:19.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:43:19.610 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:19.608+0000 7f02e3244780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:43:20.572 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:43:20.572 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:20.572 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:43:20.573 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:43:20.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:20.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:43:20.801 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1909167972,v1:127.0.0.1:6827/1909167972] [v2:127.0.0.1:6828/1909167972,v1:127.0.0.1:6829/1909167972] exists,up 76971875-5666-492d-b6fb-fadde1ccd534 2026-02-25T16:43:20.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T16:43:20.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:20.806 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:43:20.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:43:20.808 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:43:20.808 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:43:20.808 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T16:43:20.808 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:43:20.808 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=86943215-1975-463a-8cbf-5b3f4f02e2f4 2026-02-25T16:43:20.808 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 86943215-1975-463a-8cbf-5b3f4f02e2f4 2026-02-25T16:43:20.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 86943215-1975-463a-8cbf-5b3f4f02e2f4' 2026-02-25T16:43:20.809 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:43:20.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoJp9plvanMBAA0RVpx0A6uc4P34pyX/Wa4w== 2026-02-25T16:43:20.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoJp9plvanMBAA0RVpx0A6uc4P34pyX/Wa4w=="}' 2026-02-25T16:43:20.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 86943215-1975-463a-8cbf-5b3f4f02e2f4 -i td/test-erasure-eio/4/new.json 2026-02-25T16:43:21.048 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:43:21.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T16:43:21.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoJp9plvanMBAA0RVpx0A6uc4P34pyX/Wa4w== --osd-uuid 86943215-1975-463a-8cbf-5b3f4f02e2f4 2026-02-25T16:43:21.077 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.075+0000 7fe20ac3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:21.078 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.077+0000 7fe20ac3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:21.079 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.077+0000 7fe20ac3e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:21.080 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.078+0000 7fe20ac3e780 -1 bdev(0x2f14c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:43:21.080 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.078+0000 7fe20ac3e780 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T16:43:21.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T16:43:21.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:43:21.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:43:21.424 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:43:21.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:43:21.751 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:43:21.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:43:21.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:43:21.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:43:21.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:43:21.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:43:21.771 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.770+0000 7fee08e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:21.772 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.771+0000 7fee08e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:21.773 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.772+0000 7fee08e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:21.915 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:21.914+0000 7fee08e39780 -1 Falling back to public interface 2026-02-25T16:43:21.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:43:21.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:43:21.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:43:21.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:43:21.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:43:21.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:21.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:43:21.983 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:43:21.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:21.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:43:22.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:43:22.324 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:22.323+0000 7fee08e39780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:43:23.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:43:23.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:23.212 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:43:23.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:43:23.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:23.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:43:23.441 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 34 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3361312811,v1:127.0.0.1:6835/3361312811] [v2:127.0.0.1:6836/3361312811,v1:127.0.0.1:6837/3361312811] exists,up 86943215-1975-463a-8cbf-5b3f4f02e2f4 2026-02-25T16:43:23.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:43:23.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:43:23.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:43:23.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:43:23.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:43:23.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:43:23.444 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:23.444 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:23.444 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:23.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:43:23.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:43:23.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:43:23.487 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:43:23.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:43:23.493 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:43:11.667+0000 7f7c7043a780 0 load: jerasure load: lrc 2026-02-25T16:43:23.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:43:23.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:641: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2026-02-25T16:43:23.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T16:43:23.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:43:23.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:43:23.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T16:43:23.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:43:23.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T16:43:23.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:43:23.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T16:43:23.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:43:23.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:43:24.152 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:43:24.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:43:25.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:43:25.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:43:25.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:43:25.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:43:25.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:43:25.163 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:43:25.164 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:43:25.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:43:25.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:43:25.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:43:25.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:43:25.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:43:25.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:43:25.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:43:25.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:43:25.248 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:43:25.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:43:25.472 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:43:25.473 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:43:25.473 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:43:25.473 INFO:tasks.workunit.client.0.trial129.stderr:4' 2026-02-25T16:43:25.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:43:25.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:25.473 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:43:25.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T16:43:25.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T16:43:25.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T16:43:25.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:25.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:43:25.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607557 2026-02-25T16:43:25.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607557 2026-02-25T16:43:25.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557' 2026-02-25T16:43:25.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:25.603 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:43:25.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-25T16:43:25.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-25T16:43:25.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557 2-77309411332' 2026-02-25T16:43:25.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:25.666 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:43:25.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149699 2026-02-25T16:43:25.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149699 2026-02-25T16:43:25.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557 2-77309411332 3-111669149699' 2026-02-25T16:43:25.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:25.731 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:43:25.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888067 2026-02-25T16:43:25.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888067 2026-02-25T16:43:25.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-51539607557 2-77309411332 3-111669149699 4-146028888067' 2026-02-25T16:43:25.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:25.792 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T16:43:25.792 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:25.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:43:25.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T16:43:25.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:25.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T16:43:25.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T16:43:25.795 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803781 2026-02-25T16:43:25.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:43:26.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803781 2026-02-25T16:43:26.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:26.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607557 2026-02-25T16:43:26.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:26.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:43:26.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607557 2026-02-25T16:43:26.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:26.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607557 2026-02-25T16:43:26.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607557' 2026-02-25T16:43:26.022 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607557 2026-02-25T16:43:26.022 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:43:26.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607557 -lt 51539607557 2026-02-25T16:43:26.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:26.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411332 2026-02-25T16:43:26.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:26.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:43:26.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411332 2026-02-25T16:43:26.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:26.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411332 2026-02-25T16:43:26.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-25T16:43:26.252 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411332 2026-02-25T16:43:26.252 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:43:26.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411330 -lt 77309411332 2026-02-25T16:43:26.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:43:27.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:43:27.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:43:27.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411332 -lt 77309411332 2026-02-25T16:43:27.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:27.708 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149699 2026-02-25T16:43:27.709 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:27.709 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:43:27.709 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149699 2026-02-25T16:43:27.709 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:27.710 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149699 2026-02-25T16:43:27.711 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2026-02-25T16:43:27.711 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 111669149699 2026-02-25T16:43:27.711 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:43:27.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149700 -lt 111669149699 2026-02-25T16:43:27.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:27.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888067 2026-02-25T16:43:27.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:27.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:43:27.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888067 2026-02-25T16:43:27.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:27.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888067 2026-02-25T16:43:27.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888067' 2026-02-25T16:43:27.940 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 146028888067 2026-02-25T16:43:27.941 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:43:28.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888067 -lt 146028888067 2026-02-25T16:43:28.168 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:43:28.168 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:28.168 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:43:28.486 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:43:28.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:43:28.712 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:43:28.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:28.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:29.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:43:29.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:43:29.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:43:29.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:644: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:25.165567+0000 0'0 42:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T16:43:24.088849+0000 0'0 2026-02-25T16:43:24.088849+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:45:27.347421+0000 0 0 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.347166+0000 0'0 42:23 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T19:57:19.266664+0000 0 0 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.224353+0000 0'0 42:99 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:49:24.164485+0000 0 0 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.590299+0000 0'0 42:23 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-27T00:07:24.446276+0000 0 0 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:18.459483+0000 0'0 42:66 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T19:00:12.412971+0000 0 0 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout: 2026-02-25T16:43:29.238 INFO:tasks.workunit.client.0.trial129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T16:43:29.239 INFO:tasks.workunit.client.0.trial129.stderr:dumped pgs 2026-02-25T16:43:29.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:43:29.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:43:29.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:43:29.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T16:43:29.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:43:29.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:29.265 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2026-02-25T16:43:29.265 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:43:29.265 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:43:29.266 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:43:29.266 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:43:29.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:43:29.494 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:43:29.494 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:43:29.494 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:43:29.494 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:43:29.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: local -a initial_osds 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:649: TEST_ec_recovery_unfound: local last_osd=2 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:43:29.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:43:29.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:43:29.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: ceph osd down 2 2026-02-25T16:43:30.026 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already down. 2026-02-25T16:43:30.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: ceph osd out 2 2026-02-25T16:43:30.279 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already out. 2026-02-25T16:43:30.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-25T16:43:30.506 INFO:tasks.workunit.client.0.trial129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T16:43:30.506 INFO:tasks.workunit.client.0.trial129.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:25.165567+0000 0'0 42:16 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T16:43:24.088849+0000 0'0 2026-02-25T16:43:24.088849+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:45:27.347421+0000 0 0 2026-02-25T16:43:30.506 INFO:tasks.workunit.client.0.trial129.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.347166+0000 0'0 42:23 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T19:57:19.266664+0000 0 0 2026-02-25T16:43:30.506 INFO:tasks.workunit.client.0.trial129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.224353+0000 0'0 42:99 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:49:24.164485+0000 0 0 2026-02-25T16:43:30.506 INFO:tasks.workunit.client.0.trial129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.590299+0000 0'0 42:23 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-27T00:07:24.446276+0000 0 0 2026-02-25T16:43:30.506 INFO:tasks.workunit.client.0.trial129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:18.459483+0000 0'0 42:66 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T19:00:12.412971+0000 0 0 2026-02-25T16:43:30.506 INFO:tasks.workunit.client.0.trial129.stdout: 2026-02-25T16:43:30.507 INFO:tasks.workunit.client.0.trial129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T16:43:30.507 INFO:tasks.workunit.client.0.trial129.stderr:dumped pgs 2026-02-25T16:43:30.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:656: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2026-02-25T16:43:30.513 INFO:tasks.workunit.client.0.trial129.stderr:4+0 records in 2026-02-25T16:43:30.513 INFO:tasks.workunit.client.0.trial129.stderr:4+0 records out 2026-02-25T16:43:30.514 INFO:tasks.workunit.client.0.trial129.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.2516e-05 s, 56.5 MB/s 2026-02-25T16:43:30.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: seq 1 100 2026-02-25T16:43:30.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.828 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.959 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.959 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.976 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:30.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:30.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.605 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.705 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.705 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.756 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:31.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:31.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:32.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:659: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2026-02-25T16:43:32.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2026-02-25T16:43:32.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj75 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T16:43:32.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:43:32.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj75 2026-02-25T16:43:32.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:43:32.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2026-02-25T16:43:32.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2026-02-25T16:43:32.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:2147483647' 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:43:32.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T16:43:32.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:43:32.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T16:43:32.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:43:32.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T16:43:32.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:43:32.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:43:32.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:43:32.401 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T16:43:32.401 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:43:32.401 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:43:32.401 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:32.401 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:32.401 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:32.401 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:43:32.402 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:43:32.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:43:32.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:43:32.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:43:32.449 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T16:43:32.449 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:43:32.449 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:43:32.449 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:32.449 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:32.450 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:32.450 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:43:32.450 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:43:32.450 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:663: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj75 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj75 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:43:32.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj75 2026-02-25T16:43:32.498 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj75 2026-02-25T16:43:32.498 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:43:32.724 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:2147483647' 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 4 0 2147483647 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T16:43:32.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:43:32.726 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T16:43:32.727 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:43:32.727 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:43:32.727 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:32.727 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:32.727 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:32.727 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:43:32.727 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:43:32.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:43:32.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:43:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:43:32.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T16:43:32.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:43:32.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:43:32.780 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:43:32.780 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:32.780 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:32.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:43:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:43:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:665: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:32.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:43:32.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:43:32.843 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:32.841+0000 7f4a74221780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:32.844 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:32.843+0000 7f4a74221780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:32.845 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:32.843+0000 7f4a74221780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:43:32.999 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:32.997+0000 7f4a74221780 -1 Falling back to public interface 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:33.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:43:33.283 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:43:33.335 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:43:33.334+0000 7f4a74221780 -1 osd.2 42 log_to_monitors true 2026-02-25T16:43:34.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:43:34.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:43:34.285 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:43:34.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:43:34.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:43:34.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:43:34.521 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up out weight 0 up_from 48 up_thru 20 down_at 43 last_clean_interval [18,42) [v2:127.0.0.1:6818/2813108633,v1:127.0.0.1:6819/2813108633] [v2:127.0.0.1:6820/2813108633,v1:127.0.0.1:6821/2813108633] exists,up f1895396-3d26-4c78-abbe-b2398de2a830 2026-02-25T16:43:34.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:43:34.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:43:34.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:43:34.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph osd in 2 2026-02-25T16:43:34.831 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already in. 2026-02-25T16:43:34.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:668: TEST_ec_recovery_unfound: sleep 15 2026-02-25T16:43:49.841 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: seq 1 100 2026-02-25T16:43:49.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2026-02-25T16:43:49.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: get_state 2.0 2026-02-25T16:43:49.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2026-02-25T16:43:49.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2026-02-25T16:43:49.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2026-02-25T16:43:49.846 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2026-02-25T16:43:50.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:671: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2026-02-25T16:43:50.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2026-02-25T16:43:50.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:672: TEST_ec_recovery_unfound: grep recovery_unfound 2026-02-25T16:43:50.071 INFO:tasks.workunit.client.0.trial129.stdout:active+recovery_unfound+degraded 2026-02-25T16:43:50.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2026-02-25T16:43:50.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: break 2026-02-25T16:43:50.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2026-02-25T16:43:34.870003+0000 46'101 53:362 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-25T16:43:24.088849+0000 0'0 2026-02-25T16:43:24.088849+0000 0 0 no scrub is scheduled 0 0 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:34.868277+0000 0'0 53:61 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T21:05:49.537613+0000 0 0 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.224353+0000 0'0 53:118 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-26T18:49:24.164485+0000 0 0 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:23.590299+0000 0'0 53:43 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-27T00:07:24.446276+0000 0 0 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-25T16:43:34.868228+0000 0'0 53:105 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-25T16:43:09.720171+0000 0'0 2026-02-25T16:43:09.720171+0000 0 0 periodic scrub scheduled @ 2026-02-27T00:00:09.232061+0000 0 0 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout: 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-25T16:43:50.289 INFO:tasks.workunit.client.0.trial129.stderr:dumped pgs 2026-02-25T16:43:50.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:681: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout:{ 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "num_missing": 1, 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "num_unfound": 1, 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "objects": [ 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "oid": { 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "oid": "obj75", 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "key": "", 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "snapid": -2, 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "hash": 3565005477, 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "max": 0, 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "pool": 2, 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "namespace": "" 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "need": "46'76", 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "have": "0'0", 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "flags": "none", 2026-02-25T16:43:50.356 INFO:tasks.workunit.client.0.trial129.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: "locations": [ 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: ] 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: "state": "NotRecovering", 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: "available_might_have_unfound": true, 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: "might_have_unfound": [], 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout: "more": false 2026-02-25T16:43:50.357 INFO:tasks.workunit.client.0.trial129.stdout:} 2026-02-25T16:43:50.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: ceph pg 2.0 query 2026-02-25T16:43:50.422 INFO:tasks.workunit.client.0.trial129.stdout:{ 2026-02-25T16:43:50.422 INFO:tasks.workunit.client.0.trial129.stdout: "snap_trimq": "[]", 2026-02-25T16:43:50.422 INFO:tasks.workunit.client.0.trial129.stdout: "snap_trimq_len": 0, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+recovery_unfound+degraded", 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "epoch": 53, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "acting_recovery_backfill": [ 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:43:50.423 INFO:tasks.workunit.client.0.trial129.stdout: "2(4)", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "info": { 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s0", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "0", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'101", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'75", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "0'0", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 101, 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "52", 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 52, 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.424 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 52, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 52, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'101", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 363, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 53, 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+recovery_unfound+degraded", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:43:34.870003+0000", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:43:34.870003+0000", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:43:34.870003+0000", 2026-02-25T16:43:50.425 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:43:34.870003+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:43:29.262326+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:43:34.868423+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:43:34.868423+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:43:34.870003+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:43:34.867030+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:43:34.870003+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 52, 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "0'0", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "0'0", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:43:50.426 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 0, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "no scrub is scheduled", 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 413696, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 101, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 505, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 1, 2026-02-25T16:43:50.427 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 1, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 3, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 1, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 101, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 101, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 404, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:43:50.428 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.429 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 100 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),4(2)", 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 1 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:43:50.430 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "peer_info": [ 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "peer": "0(3)", 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s3", 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "3", 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'101", 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'101", 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "0'0", 2026-02-25T16:43:50.431 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 101, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 52, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 52, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 52, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.432 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'101", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 351, 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 52, 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+remapped", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:43:29.262326+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:43:29.677537+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:43:29.677469+0000", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 52, 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "0'0", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "0'0", 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:43:50.433 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 0, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:43:50.434 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T02:24:08.873111+0000", 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 413696, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 101, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 505, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 101, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 101, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 101, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 404, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:43:50.435 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:43:50.436 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:43:50.437 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 101 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "peer": "1(1)", 2026-02-25T16:43:50.438 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s1", 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "1", 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'101", 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'101", 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "0'0", 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 101, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 52, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 52, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 52, 2026-02-25T16:43:50.439 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'101", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 132, 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 46, 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+remapped", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:43:29.262326+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:43:29.677537+0000", 2026-02-25T16:43:50.440 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:43:29.677469+0000", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 52, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "0'0", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "0'0", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 0, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:43:50.441 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T02:24:08.873111+0000", 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 413696, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 101, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 505, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 1, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 101, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 101, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:43:50.442 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 101, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 404, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:43:50.443 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.444 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 101 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:43:50.445 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "peer": "2(4)", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s4", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "4", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'101", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'75", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "0'0", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 101, 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 52, 2026-02-25T16:43:50.446 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 52, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 52, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'101", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 139, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 50, 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+remapped", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:43:33.645888+0000", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:43:33.544466+0000", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:43:33.544466+0000", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:43:33.523008+0000", 2026-02-25T16:43:50.447 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:43:29.262326+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:43:33.645888+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:43:33.645888+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:43:33.645888+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 52, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "0'0", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "0'0", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 0, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:43:50.448 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "no scrub is scheduled", 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 413696, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 101, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 505, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 1, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:43:50.449 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 101, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 101, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 404, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:43:50.450 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.451 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [], 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [], 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "peer": "4(2)", 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s2", 2026-02-25T16:43:50.452 INFO:tasks.workunit.client.0.trial129.stdout: "shard": "2", 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_update": "46'101", 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_complete": "46'101", 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "log_tail": "0'0", 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_user_version": 101, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill": "MAX", 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete": [], 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "partial_writes_last_complete_epoch": "0", 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [], 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "history": { 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_created": 40, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "epoch_pool_created": 40, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_started": 52, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_interval_clean": 40, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_split": 0, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_marked_full": 0, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "same_up_since": 52, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "same_interval_since": 52, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "same_primary_since": 40, 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.453 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "prior_readable_until_ub": 0 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "stats": { 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "version": "46'101", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "reported_seq": 132, 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "reported_epoch": 46, 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "state": "active+remapped", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_fresh": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_change": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_active": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_peered": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean": "2026-02-25T16:43:29.262326+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_active": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_became_peered": "2026-02-25T16:43:29.678684+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_unstale": "2026-02-25T16:43:32.171730+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_undegraded": "2026-02-25T16:43:29.677537+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "last_fullsized": "2026-02-25T16:43:29.677469+0000", 2026-02-25T16:43:50.454 INFO:tasks.workunit.client.0.trial129.stdout: "mapping_epoch": 52, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "log_start": "0'0", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_start": "0'0", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "created": 40, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_clean": 41, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "parent": "0.0", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "parent_split_bits": 0, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub": "0'0", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub": "0'0", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "last_deep_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "last_clean_scrub_stamp": "2026-02-25T16:43:24.088849+0000", 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "objects_scrubbed": 0, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "log_size": 101, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "log_dups_size": 0, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "ondisk_log_size": 101, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "stats_invalid": false, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "dirty_stats_invalid": false, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "omap_stats_invalid": false, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_stats_invalid": false, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "hitset_bytes_stats_invalid": false, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "pin_stats_invalid": false, 2026-02-25T16:43:50.455 INFO:tasks.workunit.client.0.trial129.stdout: "manifest_stats_invalid": false, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrimq_len": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "last_scrub_duration": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-27T02:24:08.873111+0000", 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "scrub_duration": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "objects_trimmed": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "snaptrim_duration": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "stat_sum": { 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes": 413696, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects": 101, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_clones": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_object_copies": 505, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing_on_primary": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_missing": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_degraded": 101, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_misplaced": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_unfound": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_dirty": 101, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_whiteouts": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_read": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_read_kb": 0, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_write": 101, 2026-02-25T16:43:50.456 INFO:tasks.workunit.client.0.trial129.stdout: "num_write_kb": 404, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_scrub_errors": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_shallow_scrub_errors": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_deep_scrub_errors": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_recovered": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_recovered": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_keys_recovered": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_omap": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_hit_set_archive": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_bytes_hit_set_archive": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_kb": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_kb": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_promote": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_high": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_flush_mode_low": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_some": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_evict_mode_full": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_pinned": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_legacy_snapsets": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_large_omap_objects": 0, 2026-02-25T16:43:50.457 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_manifest": 0, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_bytes": 0, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "num_omap_keys": 0, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "num_objects_repaired": 0 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "up": [ 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "acting": [ 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 3, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 1, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 4, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 0, 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: 2 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "avail_no_missing": [ 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "3(0)", 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "0(3)", 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "1(1)", 2026-02-25T16:43:50.458 INFO:tasks.workunit.client.0.trial129.stdout: "4(2)" 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "object_location_counts": [ 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "objects": 101 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "blocked_by": [], 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "up_primary": 3, 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "acting_primary": 3, 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "purged_snaps": [] 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "empty": 0, 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "dne": 0, 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "incomplete": 0, 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "last_epoch_started": 53, 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "hit_set_history": { 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "current_last_update": "0'0", 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "history": [] 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: "recovery_state": [ 2026-02-25T16:43:50.459 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "name": "Started/Primary/Active", 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "enter_time": "2026-02-25T16:43:34.866750+0000", 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "might_have_unfound": [ 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "0(3)", 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "status": "already probed" 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "1(1)", 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "status": "already probed" 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "2(4)", 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "status": "already probed" 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "osd": "4(2)", 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "status": "already probed" 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "recovery_progress": { 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "backfill_targets": [], 2026-02-25T16:43:50.460 INFO:tasks.workunit.client.0.trial129.stdout: "waiting_on_backfill": [], 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "last_backfill_started": "MIN", 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "backfill_info": { 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "begin": "MIN", 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "end": "MIN", 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "objects": [] 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "peer_backfill_info": [], 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "backfills_in_flight": [], 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "recovering": [], 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "pg_backend": { 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "recovery_ops": [], 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "read_ops": [] 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: { 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "name": "Started", 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "enter_time": "2026-02-25T16:43:34.764779+0000" 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: ], 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "scrubber": { 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "active": false, 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "must_scrub": false, 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "must_deep_scrub": false, 2026-02-25T16:43:50.461 INFO:tasks.workunit.client.0.trial129.stdout: "must_repair": false, 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "sched_time": "2026-02-27T02:24:08.873111+0000", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "schedule": "not registered for scrubbing", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "urgency": "periodic-regular", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "shallow-target": { 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s0", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "level": "shallow", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "urgency": "periodic-regular", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "sched_time": "2026-02-27T02:24:08.873111+0000", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "orig_sched_time": "2026-02-27T02:24:08.873111+0000", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "last_issue": "ok", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "forced": false, 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "eligible": false, 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "queued": false, 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "active": false 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "deep-target": { 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "pgid": "2.0s0", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "level": "deep", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "urgency": "periodic-regular", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "sched_time": "2026-03-03T00:50:44.221105+0000", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "orig_sched_time": "2026-03-03T00:50:44.221105+0000", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "last_issue": "ok", 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "forced": false, 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "eligible": false, 2026-02-25T16:43:50.462 INFO:tasks.workunit.client.0.trial129.stdout: "queued": false, 2026-02-25T16:43:50.463 INFO:tasks.workunit.client.0.trial129.stdout: "active": false 2026-02-25T16:43:50.463 INFO:tasks.workunit.client.0.trial129.stdout: } 2026-02-25T16:43:50.463 INFO:tasks.workunit.client.0.trial129.stdout: }, 2026-02-25T16:43:50.463 INFO:tasks.workunit.client.0.trial129.stdout: "agent_state": {} 2026-02-25T16:43:50.463 INFO:tasks.workunit.client.0.trial129.stdout:} 2026-02-25T16:43:50.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:43:50.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: grep -q obj75 2026-02-25T16:43:50.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:43:50.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2026-02-25T16:43:50.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: check=true 2026-02-25T16:43:50.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: test true == true 2026-02-25T16:43:50.557 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-25T16:43:50.557 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2026-02-25T16:43:50.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:689: TEST_ec_recovery_unfound: check=0 2026-02-25T16:43:50.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:690: TEST_ec_recovery_unfound: test 0 == 0 2026-02-25T16:43:50.621 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-25T16:43:55.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:694: TEST_ec_recovery_unfound: test 124 = 124 2026-02-25T16:43:55.625 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:696: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2026-02-25T16:43:55.689 INFO:tasks.workunit.client.0.trial129.stderr:pg has no unfound objects 2026-02-25T16:43:55.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:698: TEST_ec_recovery_unfound: wait_for_clean 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:43:55.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:43:55.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:43:55.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:43:55.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:43:55.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:43:55.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:43:55.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:43:55.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr:4' 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:56.006 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:43:56.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803789 2026-02-25T16:43:56.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803789 2026-02-25T16:43:56.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789' 2026-02-25T16:43:56.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:56.072 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:43:56.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607565 2026-02-25T16:43:56.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607565 2026-02-25T16:43:56.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-51539607565' 2026-02-25T16:43:56.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:56.137 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:43:56.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=206158430215 2026-02-25T16:43:56.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 206158430215 2026-02-25T16:43:56.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-51539607565 2-206158430215' 2026-02-25T16:43:56.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:56.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:43:56.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149708 2026-02-25T16:43:56.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149708 2026-02-25T16:43:56.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-51539607565 2-206158430215 3-111669149708' 2026-02-25T16:43:56.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:43:56.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:43:56.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888075 2026-02-25T16:43:56.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888075 2026-02-25T16:43:56.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803789 1-51539607565 2-206158430215 3-111669149708 4-146028888075' 2026-02-25T16:43:56.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:56.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803789 2026-02-25T16:43:56.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:56.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:43:56.334 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803789 2026-02-25T16:43:56.334 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:56.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803789 2026-02-25T16:43:56.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2026-02-25T16:43:56.335 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803789 2026-02-25T16:43:56.335 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:43:56.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803789 2026-02-25T16:43:56.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:43:57.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:43:57.565 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:43:57.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803789 -lt 25769803789 2026-02-25T16:43:57.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:57.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607565 2026-02-25T16:43:57.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:57.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:43:57.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607565 2026-02-25T16:43:57.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:57.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607565 2026-02-25T16:43:57.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607565' 2026-02-25T16:43:57.796 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607565 2026-02-25T16:43:57.797 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:43:58.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607565 -lt 51539607565 2026-02-25T16:43:58.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:58.025 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-206158430215 2026-02-25T16:43:58.025 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:58.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:43:58.026 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-206158430215 2026-02-25T16:43:58.026 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:58.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=206158430215 2026-02-25T16:43:58.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 206158430215' 2026-02-25T16:43:58.028 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 206158430215 2026-02-25T16:43:58.028 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:43:58.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 206158430215 -lt 206158430215 2026-02-25T16:43:58.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:58.256 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149708 2026-02-25T16:43:58.256 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:58.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:43:58.257 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149708 2026-02-25T16:43:58.258 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:58.258 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149708 2026-02-25T16:43:58.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149708' 2026-02-25T16:43:58.259 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 111669149708 2026-02-25T16:43:58.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:43:58.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149708 -lt 111669149708 2026-02-25T16:43:58.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:43:58.488 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-146028888075 2026-02-25T16:43:58.488 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:43:58.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:43:58.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-146028888075 2026-02-25T16:43:58.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:43:58.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=146028888075 2026-02-25T16:43:58.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 146028888075' 2026-02-25T16:43:58.492 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 146028888075 2026-02-25T16:43:58.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:43:58.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 146028888075 -lt 146028888075 2026-02-25T16:43:58.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:43:58.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:58.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:59.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:43:59.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:43:59.043 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:43:59.043 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:43:59.043 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:43:59.043 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:43:59.044 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:43:59.044 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:43:59.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:43:59.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:43:59.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:43:59.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:43:59.581 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:43:59.581 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:43:59.581 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:43:59.581 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: seq 1 100 2026-02-25T16:43:59.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2026-02-25T16:43:59.582 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2026-02-25T16:43:59.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2026-02-25T16:43:59.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2026-02-25T16:43:59.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2026-02-25T16:43:59.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.668 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.668 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2026-02-25T16:43:59.668 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.684 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2026-02-25T16:43:59.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2026-02-25T16:43:59.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2026-02-25T16:43:59.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2026-02-25T16:43:59.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2026-02-25T16:43:59.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.766 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.767 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2026-02-25T16:43:59.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2026-02-25T16:43:59.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2026-02-25T16:43:59.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2026-02-25T16:43:59.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2026-02-25T16:43:59.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2026-02-25T16:43:59.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.868 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2026-02-25T16:43:59.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2026-02-25T16:43:59.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2026-02-25T16:43:59.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.920 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.920 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2026-02-25T16:43:59.920 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2026-02-25T16:43:59.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2026-02-25T16:43:59.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.971 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2026-02-25T16:43:59.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2026-02-25T16:43:59.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:43:59.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:43:59.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2026-02-25T16:43:59.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2026-02-25T16:44:00.006 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2026-02-25T16:44:00.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2026-02-25T16:44:00.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.056 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2026-02-25T16:44:00.057 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2026-02-25T16:44:00.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2026-02-25T16:44:00.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2026-02-25T16:44:00.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2026-02-25T16:44:00.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2026-02-25T16:44:00.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2026-02-25T16:44:00.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2026-02-25T16:44:00.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2026-02-25T16:44:00.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2026-02-25T16:44:00.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2026-02-25T16:44:00.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2026-02-25T16:44:00.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2026-02-25T16:44:00.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2026-02-25T16:44:00.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2026-02-25T16:44:00.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2026-02-25T16:44:00.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2026-02-25T16:44:00.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2026-02-25T16:44:00.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.355 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.356 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.356 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2026-02-25T16:44:00.356 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2026-02-25T16:44:00.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.387 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2026-02-25T16:44:00.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2026-02-25T16:44:00.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2026-02-25T16:44:00.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2026-02-25T16:44:00.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2026-02-25T16:44:00.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2026-02-25T16:44:00.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.487 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2026-02-25T16:44:00.488 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2026-02-25T16:44:00.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2026-02-25T16:44:00.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2026-02-25T16:44:00.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2026-02-25T16:44:00.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2026-02-25T16:44:00.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.586 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.586 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2026-02-25T16:44:00.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.602 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2026-02-25T16:44:00.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.619 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.620 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.620 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2026-02-25T16:44:00.620 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2026-02-25T16:44:00.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2026-02-25T16:44:00.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.669 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2026-02-25T16:44:00.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2026-02-25T16:44:00.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2026-02-25T16:44:00.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2026-02-25T16:44:00.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2026-02-25T16:44:00.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2026-02-25T16:44:00.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.772 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2026-02-25T16:44:00.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2026-02-25T16:44:00.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2026-02-25T16:44:00.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.821 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2026-02-25T16:44:00.822 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:704: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.837 INFO:tasks.workunit.client.0.trial129.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2026-02-25T16:44:00.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2026-02-25T16:44:00.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2026-02-25T16:44:00.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.870 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2026-02-25T16:44:00.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2026-02-25T16:44:00.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.903 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2026-02-25T16:44:00.904 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.920 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2026-02-25T16:44:00.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2026-02-25T16:44:00.936 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2026-02-25T16:44:00.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2026-02-25T16:44:00.970 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2026-02-25T16:44:00.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:00.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:00.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2026-02-25T16:44:00.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2026-02-25T16:44:01.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2026-02-25T16:44:01.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2026-02-25T16:44:01.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.054 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.055 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.055 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2026-02-25T16:44:01.055 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2026-02-25T16:44:01.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2026-02-25T16:44:01.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2026-02-25T16:44:01.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2026-02-25T16:44:01.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2026-02-25T16:44:01.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2026-02-25T16:44:01.156 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2026-02-25T16:44:01.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2026-02-25T16:44:01.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2026-02-25T16:44:01.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2026-02-25T16:44:01.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:700: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-25T16:44:01.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:702: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2026-02-25T16:44:01.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:706: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2026-02-25T16:44:01.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:707: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.255 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:711: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-25T16:44:01.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:713: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:44:01.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:44:01.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:44:01.530 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:44:01.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:44:01.838 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:44:01.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:44:01.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:44:01.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:44:01.955 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:44:01.955 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:44:01.956 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:44:01.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:44:01.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:44:01.957 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:44:01.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:01.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:44:01.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:44:01.959 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:01.959 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:44:01.960 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:44:01.960 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:44:01.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:44:01.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:01.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:01.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:44:01.981 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:44:01.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:44:01.984 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:44:01.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:44:01.985 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:44:01.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:44:01.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:44:01.986 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:44:01.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:01.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:44:01.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:44:01.988 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:01.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:44:01.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:44:01.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:44:01.991 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:44:01.991 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:01.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:01.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:44:01.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:44:01.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:44:01.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:44:01.993 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:44:01.993 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:01.993 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:01.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:44:01.995 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:44:01.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:44:01.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:44:01.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:44:01.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:44:01.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:44:01.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:44:01.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:44:01.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:44:01.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:44:01.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:44:02.014 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:44:02.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:02.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:02.015 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:02.015 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:02.015 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:02.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:02.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:44:02.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:02.039 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:44:02.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:44:02.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:44:02.084 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:44:02.084 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:44:02.084 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:44:02.084 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:44:02.084 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:44:02.084 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:44:02.084 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:44:02.085 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:44:02.085 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:02.085 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:02.085 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:02.085 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:44:02.085 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:44:02.085 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:44:02.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:44:02.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:44:02.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:44:02.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:44:02.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:44:02.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:44:02.128 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:44:02.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:44:02.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:02.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:02.227 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:02.227 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:02.227 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:02.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:02.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:44:02.228 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:44:02.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:44:02.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:44:02.338 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:44:02.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:44:03.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:44:03.387 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:44:03.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:44:03.392 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:44:02.029+0000 7f4518304c00 0 load: jerasure load: lrc 2026-02-25T16:44:03.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2026-02-25T16:44:03.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:410: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2026-02-25T16:44:03.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: local objname=myobject 2026-02-25T16:44:03.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: setup_osds 7 2026-02-25T16:44:03.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-25T16:44:03.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:44:03.393 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-25T16:44:03.394 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:03.395 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:03.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:44:03.397 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:03.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c68d33c-d18d-4df1-9644-be3055591f4d 2026-02-25T16:44:03.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1c68d33c-d18d-4df1-9644-be3055591f4d' 2026-02-25T16:44:03.398 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 1c68d33c-d18d-4df1-9644-be3055591f4d 2026-02-25T16:44:03.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:03.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDTJp9pZ3RJGBAAn0r0SisnU7AGanK3y8NxsA== 2026-02-25T16:44:03.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDTJp9pZ3RJGBAAn0r0SisnU7AGanK3y8NxsA=="}' 2026-02-25T16:44:03.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c68d33c-d18d-4df1-9644-be3055591f4d -i td/test-erasure-eio/0/new.json 2026-02-25T16:44:03.510 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:03.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:44:03.517 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTJp9pZ3RJGBAAn0r0SisnU7AGanK3y8NxsA== --osd-uuid 1c68d33c-d18d-4df1-9644-be3055591f4d 2026-02-25T16:44:03.532 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:03.531+0000 7f082e1d6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:03.534 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:03.532+0000 7f082e1d6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:03.535 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:03.533+0000 7f082e1d6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:03.535 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:03.534+0000 7f082e1d6780 -1 bdev(0x2bdac00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:03.535 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:03.534+0000 7f082e1d6780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:44:04.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:44:04.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:04.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:44:04.004 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:44:04.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:04.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:44:04.108 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:44:04.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:04.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:04.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:04.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:04.123 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:04.122+0000 7fe07d613780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:04.125 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:04.123+0000 7fe07d613780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:04.126 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:04.124+0000 7fe07d613780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:04.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:44:04.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:04.206 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:44:04.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:04.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:04.207 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:04.207 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:04.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:04.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:04.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:44:04.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:04.341 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:04.339+0000 7fe07d613780 -1 Falling back to public interface 2026-02-25T16:44:04.601 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:04.599+0000 7fe07d613780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:44:05.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:05.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:05.306 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:05.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:05.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:05.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:44:05.534 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/405885438,v1:127.0.0.1:6803/405885438] [v2:127.0.0.1:6804/405885438,v1:127.0.0.1:6805/405885438] exists,up 1c68d33c-d18d-4df1-9644-be3055591f4d 2026-02-25T16:44:05.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:05.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:05.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:05.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:05.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:05.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:05.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:44:05.542 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:05.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=07afcb1f-8921-4395-b97a-760308a11691 2026-02-25T16:44:05.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 07afcb1f-8921-4395-b97a-760308a11691' 2026-02-25T16:44:05.542 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 07afcb1f-8921-4395-b97a-760308a11691 2026-02-25T16:44:05.542 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:05.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVJp9prbaMIBAAZJpzYTr2Re5wPSkXmcKukA== 2026-02-25T16:44:05.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVJp9prbaMIBAAZJpzYTr2Re5wPSkXmcKukA=="}' 2026-02-25T16:44:05.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 07afcb1f-8921-4395-b97a-760308a11691 -i td/test-erasure-eio/1/new.json 2026-02-25T16:44:05.778 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:05.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:44:05.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVJp9prbaMIBAAZJpzYTr2Re5wPSkXmcKukA== --osd-uuid 07afcb1f-8921-4395-b97a-760308a11691 2026-02-25T16:44:05.802 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:05.800+0000 7f31969df780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:05.803 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:05.802+0000 7f31969df780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:05.804 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:05.803+0000 7f31969df780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:05.805 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:05.803+0000 7f31969df780 -1 bdev(0x39aec00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:05.805 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:05.803+0000 7f31969df780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:44:06.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:44:06.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:06.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:44:06.159 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:44:06.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:06.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:44:06.473 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:44:06.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:06.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:06.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:06.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:06.488 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:06.486+0000 7fbdc67e3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:06.489 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:06.487+0000 7fbdc67e3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:06.490 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:06.488+0000 7fbdc67e3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:06.659 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:06.658+0000 7fbdc67e3780 -1 Falling back to public interface 2026-02-25T16:44:06.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:44:06.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:06.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:44:06.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:06.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:06.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:06.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:06.703 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:06.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:06.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:44:06.929 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:06.963 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:06.961+0000 7fbdc67e3780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:44:07.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:07.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:07.932 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:07.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:07.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:07.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:44:08.158 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/74464404,v1:127.0.0.1:6811/74464404] [v2:127.0.0.1:6812/74464404,v1:127.0.0.1:6813/74464404] exists,up 07afcb1f-8921-4395-b97a-760308a11691 2026-02-25T16:44:08.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:08.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:08.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:08.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:08.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:44:08.165 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:08.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4ac1bd2d-ba0e-47c1-9692-7d6098d1924e 2026-02-25T16:44:08.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4ac1bd2d-ba0e-47c1-9692-7d6098d1924e' 2026-02-25T16:44:08.166 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 4ac1bd2d-ba0e-47c1-9692-7d6098d1924e 2026-02-25T16:44:08.166 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:08.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDYJp9pIj4rChAAcGhPaRCff7n7ql+LcVpxaA== 2026-02-25T16:44:08.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDYJp9pIj4rChAAcGhPaRCff7n7ql+LcVpxaA=="}' 2026-02-25T16:44:08.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4ac1bd2d-ba0e-47c1-9692-7d6098d1924e -i td/test-erasure-eio/2/new.json 2026-02-25T16:44:08.404 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:44:08.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:44:08.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYJp9pIj4rChAAcGhPaRCff7n7ql+LcVpxaA== --osd-uuid 4ac1bd2d-ba0e-47c1-9692-7d6098d1924e 2026-02-25T16:44:08.427 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:08.425+0000 7f4a2d252780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:08.428 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:08.427+0000 7f4a2d252780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:08.429 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:08.428+0000 7f4a2d252780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:08.430 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:08.428+0000 7f4a2d252780 -1 bdev(0x4194c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:08.430 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:08.428+0000 7f4a2d252780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:44:08.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:44:08.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:08.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:44:08.789 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:44:08.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:09.105 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:44:09.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:44:09.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:09.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:09.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:09.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:09.119 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:09.117+0000 7f0ee95fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:09.120 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:09.119+0000 7f0ee95fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:09.121 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:09.120+0000 7f0ee95fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:09.250 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:09.248+0000 7f0ee95fb780 -1 Falling back to public interface 2026-02-25T16:44:09.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:44:09.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:09.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:44:09.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:09.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:09.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:09.332 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:09.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:09.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:09.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:44:09.551 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:09.550+0000 7f0ee95fb780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:44:09.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:10.561 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:10.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:44:10.784 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2369675616,v1:127.0.0.1:6819/2369675616] [v2:127.0.0.1:6820/2369675616,v1:127.0.0.1:6821/2369675616] exists,up 4ac1bd2d-ba0e-47c1-9692-7d6098d1924e 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:44:10.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:10.786 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:44:10.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:10.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c56aa139-99fa-42a0-9308-d118e6cf6060 2026-02-25T16:44:10.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 c56aa139-99fa-42a0-9308-d118e6cf6060' 2026-02-25T16:44:10.788 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 c56aa139-99fa-42a0-9308-d118e6cf6060 2026-02-25T16:44:10.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:10.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDaJp9pEGqNLxAA7PRdjAWTtMcGV2zjHB2S9g== 2026-02-25T16:44:10.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDaJp9pEGqNLxAA7PRdjAWTtMcGV2zjHB2S9g=="}' 2026-02-25T16:44:10.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c56aa139-99fa-42a0-9308-d118e6cf6060 -i td/test-erasure-eio/3/new.json 2026-02-25T16:44:11.026 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:44:11.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:44:11.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaJp9pEGqNLxAA7PRdjAWTtMcGV2zjHB2S9g== --osd-uuid c56aa139-99fa-42a0-9308-d118e6cf6060 2026-02-25T16:44:11.049 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.047+0000 7fcfc1e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:11.050 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.049+0000 7fcfc1e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:11.051 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.049+0000 7fcfc1e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:11.051 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.050+0000 7fcfc1e2c780 -1 bdev(0x4372c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:11.051 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.050+0000 7fcfc1e2c780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:44:11.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:44:11.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:11.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:44:11.423 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:44:11.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:11.738 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:44:11.739 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:44:11.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:11.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:11.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:11.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:11.752 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.751+0000 7f04f2dd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:11.754 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.752+0000 7f04f2dd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:11.755 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.753+0000 7f04f2dd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:11.912 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:11.911+0000 7f04f2dd4780 -1 Falling back to public interface 2026-02-25T16:44:11.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:44:11.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:11.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:44:11.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:11.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:11.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:11.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:11.967 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:11.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:11.967 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:44:12.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:12.225 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:12.223+0000 7f04f2dd4780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:44:13.192 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:13.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:13.193 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:13.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:13.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:13.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:44:13.417 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78843613,v1:127.0.0.1:6827/78843613] [v2:127.0.0.1:6828/78843613,v1:127.0.0.1:6829/78843613] exists,up c56aa139-99fa-42a0-9308-d118e6cf6060 2026-02-25T16:44:13.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:13.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:13.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:13.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:13.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-25T16:44:13.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:13.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:13.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:13.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1ce62601-a4b0-4b70-a2a0-eb7042d0f780 2026-02-25T16:44:13.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 1ce62601-a4b0-4b70-a2a0-eb7042d0f780' 2026-02-25T16:44:13.426 INFO:tasks.workunit.client.0.trial129.stdout:add osd4 1ce62601-a4b0-4b70-a2a0-eb7042d0f780 2026-02-25T16:44:13.426 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:13.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDdJp9pOtejGRAAMffue+TPVZfxHWXt5yMQXg== 2026-02-25T16:44:13.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDdJp9pOtejGRAAMffue+TPVZfxHWXt5yMQXg=="}' 2026-02-25T16:44:13.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1ce62601-a4b0-4b70-a2a0-eb7042d0f780 -i td/test-erasure-eio/4/new.json 2026-02-25T16:44:13.660 INFO:tasks.workunit.client.0.trial129.stdout:4 2026-02-25T16:44:13.666 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-25T16:44:13.667 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdJp9pOtejGRAAMffue+TPVZfxHWXt5yMQXg== --osd-uuid 1ce62601-a4b0-4b70-a2a0-eb7042d0f780 2026-02-25T16:44:13.682 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:13.680+0000 7f6ac781c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:13.683 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:13.682+0000 7f6ac781c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:13.684 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:13.683+0000 7f6ac781c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:13.685 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:13.683+0000 7f6ac781c780 -1 bdev(0x2496c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:13.685 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:13.683+0000 7f6ac781c780 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-25T16:44:14.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-25T16:44:14.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:14.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-25T16:44:14.022 INFO:tasks.workunit.client.0.trial129.stdout:adding osd4 key to auth repository 2026-02-25T16:44:14.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:14.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-25T16:44:14.339 INFO:tasks.workunit.client.0.trial129.stdout:start osd.4 2026-02-25T16:44:14.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:14.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:14.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:14.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:14.352 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:14.351+0000 7fbee283c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:14.354 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:14.352+0000 7fbee283c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:14.354 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:14.353+0000 7fbee283c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:14.528 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:14.526+0000 7fbee283c780 -1 Falling back to public interface 2026-02-25T16:44:14.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-25T16:44:14.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:14.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-25T16:44:14.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:14.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:14.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:14.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:14.565 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:14.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:14.566 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:44:14.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:14.856 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:14.855+0000 7fbee283c780 -1 osd.4 0 log_to_monitors true 2026-02-25T16:44:15.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:15.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:15.788 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:15.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:15.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:15.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-25T16:44:16.013 INFO:tasks.workunit.client.0.trial129.stdout:osd.4 up in weight 1 up_from 35 up_thru 36 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/684491608,v1:127.0.0.1:6835/684491608] [v2:127.0.0.1:6836/684491608,v1:127.0.0.1:6837/684491608] exists,up 1ce62601-a4b0-4b70-a2a0-eb7042d0f780 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:16.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:16.018 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:16.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4ad12ec3-ce27-4480-ae86-80a1248b6df2 2026-02-25T16:44:16.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 4ad12ec3-ce27-4480-ae86-80a1248b6df2' 2026-02-25T16:44:16.021 INFO:tasks.workunit.client.0.trial129.stdout:add osd5 4ad12ec3-ce27-4480-ae86-80a1248b6df2 2026-02-25T16:44:16.021 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:16.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDgJp9p98SbARAAJvGd5zbLYeAeU4d2OawM5g== 2026-02-25T16:44:16.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDgJp9p98SbARAAJvGd5zbLYeAeU4d2OawM5g=="}' 2026-02-25T16:44:16.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4ad12ec3-ce27-4480-ae86-80a1248b6df2 -i td/test-erasure-eio/5/new.json 2026-02-25T16:44:16.258 INFO:tasks.workunit.client.0.trial129.stdout:5 2026-02-25T16:44:16.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-25T16:44:16.266 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgJp9p98SbARAAJvGd5zbLYeAeU4d2OawM5g== --osd-uuid 4ad12ec3-ce27-4480-ae86-80a1248b6df2 2026-02-25T16:44:16.282 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.280+0000 7f2e1e7fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:16.283 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.282+0000 7f2e1e7fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:16.284 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.283+0000 7f2e1e7fb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:16.285 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.283+0000 7f2e1e7fb780 -1 bdev(0x337ec00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:16.285 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.283+0000 7f2e1e7fb780 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-25T16:44:16.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-25T16:44:16.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:16.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-25T16:44:16.633 INFO:tasks.workunit.client.0.trial129.stdout:adding osd5 key to auth repository 2026-02-25T16:44:16.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:16.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-25T16:44:16.948 INFO:tasks.workunit.client.0.trial129.stdout:start osd.5 2026-02-25T16:44:16.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:16.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:16.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:16.949 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:16.963 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.961+0000 7fb2b6e34780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:16.964 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.963+0000 7fb2b6e34780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:16.965 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:16.964+0000 7fb2b6e34780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:17.135 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:17.133+0000 7fb2b6e34780 -1 Falling back to public interface 2026-02-25T16:44:17.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-25T16:44:17.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:17.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-25T16:44:17.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:17.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:17.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:17.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:17.176 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:17.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:17.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:44:17.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:17.478 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:17.476+0000 7fb2b6e34780 -1 osd.5 0 log_to_monitors true 2026-02-25T16:44:18.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:18.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:18.404 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:18.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:18.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:18.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-25T16:44:18.629 INFO:tasks.workunit.client.0.trial129.stdout:osd.5 up in weight 1 up_from 43 up_thru 45 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1352766023,v1:127.0.0.1:6843/1352766023] [v2:127.0.0.1:6844/1352766023,v1:127.0.0.1:6845/1352766023] exists,up 4ad12ec3-ce27-4480-ae86-80a1248b6df2 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:18.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:18.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:18.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-25T16:44:18.633 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:18.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c419e5cd-da9b-4e86-84af-cb97e455d5db 2026-02-25T16:44:18.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 c419e5cd-da9b-4e86-84af-cb97e455d5db' 2026-02-25T16:44:18.633 INFO:tasks.workunit.client.0.trial129.stdout:add osd6 c419e5cd-da9b-4e86-84af-cb97e455d5db 2026-02-25T16:44:18.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:18.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDiJp9pDdRKJhAABMmz8uBjxewzNVPT3RnItg== 2026-02-25T16:44:18.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDiJp9pDdRKJhAABMmz8uBjxewzNVPT3RnItg=="}' 2026-02-25T16:44:18.644 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c419e5cd-da9b-4e86-84af-cb97e455d5db -i td/test-erasure-eio/6/new.json 2026-02-25T16:44:18.880 INFO:tasks.workunit.client.0.trial129.stdout:6 2026-02-25T16:44:18.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-25T16:44:18.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDiJp9pDdRKJhAABMmz8uBjxewzNVPT3RnItg== --osd-uuid c419e5cd-da9b-4e86-84af-cb97e455d5db 2026-02-25T16:44:18.905 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:18.903+0000 7fc526a0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:18.907 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:18.905+0000 7fc526a0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:18.907 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:18.906+0000 7fc526a0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:18.908 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:18.906+0000 7fc526a0e780 -1 bdev(0x2460c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:18.908 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:18.906+0000 7fc526a0e780 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-25T16:44:19.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-25T16:44:19.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:19.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-25T16:44:19.272 INFO:tasks.workunit.client.0.trial129.stdout:adding osd6 key to auth repository 2026-02-25T16:44:19.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:19.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-25T16:44:19.593 INFO:tasks.workunit.client.0.trial129.stdout:start osd.6 2026-02-25T16:44:19.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:19.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:19.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:19.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:19.607 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:19.606+0000 7f61ad035780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:19.609 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:19.607+0000 7f61ad035780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:19.610 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:19.608+0000 7f61ad035780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:19.764 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:19.762+0000 7f61ad035780 -1 Falling back to public interface 2026-02-25T16:44:19.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-25T16:44:19.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:19.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-25T16:44:19.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:19.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:19.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:19.816 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:19.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:19.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:19.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:44:20.038 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:20.133 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:20.132+0000 7f61ad035780 -1 osd.6 0 log_to_monitors true 2026-02-25T16:44:21.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:21.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:21.040 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:21.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:21.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:21.040 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-25T16:44:21.262 INFO:tasks.workunit.client.0.trial129.stdout:osd.6 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3952082436,v1:127.0.0.1:6851/3952082436] [v2:127.0.0.1:6852/3952082436,v1:127.0.0.1:6853/3952082436] exists,up c419e5cd-da9b-4e86-84af-cb97e455d5db 2026-02-25T16:44:21.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:21.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:21.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:21.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:44:21.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:44:21.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:44:21.263 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:21.263 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:21.263 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:21.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:44:21.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:44:21.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:44:21.308 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:44:21.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:44:21.314 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:44:04.342+0000 7fe07d613780 0 load: jerasure load: lrc 2026-02-25T16:44:21.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:415: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:416: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:44:21.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-25T16:44:21.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:44:21.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:44:21.972 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:44:21.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:44:22.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:44:22.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:44:22.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:44:22.982 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:44:22.982 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:44:22.983 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:44:22.984 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:44:22.984 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:44:22.984 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:44:22.984 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:44:23.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:44:23.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:44:23.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:44:23.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:44:23.074 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:44:23.074 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:23.301 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:44:23.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803782 2026-02-25T16:44:23.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803782 2026-02-25T16:44:23.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782' 2026-02-25T16:44:23.366 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:23.366 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:44:23.427 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-25T16:44:23.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-25T16:44:23.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854' 2026-02-25T16:44:23.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:23.428 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:44:23.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378630 2026-02-25T16:44:23.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378630 2026-02-25T16:44:23.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378630' 2026-02-25T16:44:23.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:23.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:44:23.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116997 2026-02-25T16:44:23.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116997 2026-02-25T16:44:23.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378630 3-115964116997' 2026-02-25T16:44:23.555 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:23.555 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:44:23.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855364 2026-02-25T16:44:23.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855364 2026-02-25T16:44:23.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378630 3-115964116997 4-150323855364' 2026-02-25T16:44:23.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:23.617 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:44:23.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-25T16:44:23.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-25T16:44:23.683 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378630 3-115964116997 4-150323855364 5-184683593732' 2026-02-25T16:44:23.683 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:23.683 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:44:23.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299395 2026-02-25T16:44:23.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299395 2026-02-25T16:44:23.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803782 1-55834574854 2-81604378630 3-115964116997 4-150323855364 5-184683593732 6-223338299395' 2026-02-25T16:44:23.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:23.745 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803782 2026-02-25T16:44:23.746 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:23.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:44:23.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803782 2026-02-25T16:44:23.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:23.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803782 2026-02-25T16:44:23.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803782' 2026-02-25T16:44:23.748 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803782 2026-02-25T16:44:23.749 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:23.972 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803780 -lt 25769803782 2026-02-25T16:44:23.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:44:24.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:44:24.973 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:25.201 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803782 2026-02-25T16:44:25.201 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:25.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574854 2026-02-25T16:44:25.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:25.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:44:25.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574854 2026-02-25T16:44:25.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:25.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574854 2026-02-25T16:44:25.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574854' 2026-02-25T16:44:25.205 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574854 2026-02-25T16:44:25.205 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:44:25.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574854 -lt 55834574854 2026-02-25T16:44:25.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:25.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378630 2026-02-25T16:44:25.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:25.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:44:25.431 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378630 2026-02-25T16:44:25.431 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:25.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378630 2026-02-25T16:44:25.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378630' 2026-02-25T16:44:25.433 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378630 2026-02-25T16:44:25.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:44:25.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378630 -lt 81604378630 2026-02-25T16:44:25.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:25.657 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116997 2026-02-25T16:44:25.657 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:25.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:44:25.658 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116997 2026-02-25T16:44:25.659 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:25.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116997 2026-02-25T16:44:25.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116997' 2026-02-25T16:44:25.660 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116997 2026-02-25T16:44:25.660 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:44:25.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-25T16:44:25.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:25.884 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855364 2026-02-25T16:44:25.884 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:25.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:44:25.885 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855364 2026-02-25T16:44:25.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:25.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855364 2026-02-25T16:44:25.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855364' 2026-02-25T16:44:25.887 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855364 2026-02-25T16:44:25.887 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:44:26.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855364 -lt 150323855364 2026-02-25T16:44:26.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:26.113 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-184683593732 2026-02-25T16:44:26.113 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:26.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:44:26.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-184683593732 2026-02-25T16:44:26.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:26.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593732 2026-02-25T16:44:26.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 184683593732' 2026-02-25T16:44:26.116 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 184683593732 2026-02-25T16:44:26.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:44:26.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-25T16:44:26.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:26.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 6-223338299395 2026-02-25T16:44:26.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:26.343 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=6 2026-02-25T16:44:26.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-223338299395 2026-02-25T16:44:26.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:26.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=223338299395 2026-02-25T16:44:26.345 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.6 seq 223338299395' 2026-02-25T16:44:26.346 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.6 seq 223338299395 2026-02-25T16:44:26.346 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-25T16:44:26.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 223338299395 -lt 223338299395 2026-02-25T16:44:26.569 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:44:26.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:26.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:26.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:44:26.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:44:26.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:44:26.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:44:26.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:44:26.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:44:26.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:44:26.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:44:27.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:44:27.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:44:27.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:27.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:27.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:44:27.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:44:27.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:44:27.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:44:27.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:44:27.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:44:27.422 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:27.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:44:27.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-25T16:44:27.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-25T16:44:27.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=myobject 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:44:27.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T16:44:27.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:44:27.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure myobject 2026-02-25T16:44:27.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:44:27.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:44:27.441 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:44:27.441 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:44:27.664 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T16:44:27.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:44:27.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T16:44:27.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:44:27.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T16:44:27.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:44:27.665 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:44:27.665 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:44:27.666 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T16:44:27.666 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:44:27.666 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:44:27.666 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:27.666 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:27.666 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:27.667 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:44:27.667 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:44:27.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:44:27.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:44:27.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:44:27.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T16:44:27.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:44:27.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:44:27.717 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:27.717 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:27.717 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:27.717 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:44:27.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:44:27.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-25T16:44:27.765 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2026-02-25T16:44:27.765 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:44:27.765 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=myobject 2026-02-25T16:44:27.766 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-25T16:44:27.766 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:44:27.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:44:27.999 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:44:27.999 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:27.999 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:44:27.999 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:44:27.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 1 0 6 2026-02-25T16:44:27.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2026-02-25T16:44:28.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_single_recovery_error: local -a initial_osds 2026-02-25T16:44:28.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:422: TEST_ec_single_recovery_error: local last_osd=6 2026-02-25T16:44:28.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2026-02-25T16:44:28.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:44:28.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:44:28.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:44:28.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:44:28.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:44:28.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:44:28.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:425: TEST_ec_single_recovery_error: ceph osd down 6 2026-02-25T16:44:28.529 INFO:tasks.workunit.client.0.trial129.stderr:osd.6 is already down. 2026-02-25T16:44:28.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_single_recovery_error: ceph osd out 6 2026-02-25T16:44:28.783 INFO:tasks.workunit.client.0.trial129.stderr:osd.6 is already out. 2026-02-25T16:44:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:429: TEST_ec_single_recovery_error: wait_for_clean 2026-02-25T16:44:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:44:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:44:28.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:44:28.791 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:44:28.792 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:44:28.792 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:44:28.792 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:44:28.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:44:28.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:44:28.878 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:44:28.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:44:28.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:44:28.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:44:28.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:44:28.879 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:44:29.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:44:29.100 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:29.100 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:44:29.100 INFO:tasks.workunit.client.0.trial129.stderr:3 2026-02-25T16:44:29.101 INFO:tasks.workunit.client.0.trial129.stderr:4 2026-02-25T16:44:29.101 INFO:tasks.workunit.client.0.trial129.stderr:5 2026-02-25T16:44:29.101 INFO:tasks.workunit.client.0.trial129.stderr:6' 2026-02-25T16:44:29.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:44:29.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:29.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:44:29.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T16:44:29.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T16:44:29.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T16:44:29.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:29.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:44:29.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-25T16:44:29.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-25T16:44:29.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857' 2026-02-25T16:44:29.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:29.232 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:44:29.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378633 2026-02-25T16:44:29.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378633 2026-02-25T16:44:29.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378633' 2026-02-25T16:44:29.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:29.298 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:44:29.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117000 2026-02-25T16:44:29.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117000 2026-02-25T16:44:29.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378633 3-115964117000' 2026-02-25T16:44:29.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:29.359 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-25T16:44:29.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855367 2026-02-25T16:44:29.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855367 2026-02-25T16:44:29.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378633 3-115964117000 4-150323855367' 2026-02-25T16:44:29.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:29.425 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-25T16:44:29.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593735 2026-02-25T16:44:29.489 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593735 2026-02-25T16:44:29.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-81604378633 3-115964117000 4-150323855367 5-184683593735' 2026-02-25T16:44:29.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:29.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-25T16:44:29.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T16:44:29.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T16:44:29.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T16:44:29.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T16:44:29.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:29.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T16:44:29.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:29.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:44:29.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T16:44:29.536 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:29.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T16:44:29.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T16:44:29.537 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803785 2026-02-25T16:44:29.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:29.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803785 2026-02-25T16:44:29.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:44:30.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:44:30.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:30.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T16:44:30.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:30.996 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574857 2026-02-25T16:44:30.996 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:30.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:44:30.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574857 2026-02-25T16:44:30.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:30.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574857 2026-02-25T16:44:30.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574857' 2026-02-25T16:44:30.999 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574857 2026-02-25T16:44:30.999 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:44:31.225 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574857 -lt 55834574857 2026-02-25T16:44:31.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:31.226 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378633 2026-02-25T16:44:31.226 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:31.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:44:31.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378633 2026-02-25T16:44:31.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:31.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378633 2026-02-25T16:44:31.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378633' 2026-02-25T16:44:31.229 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378633 2026-02-25T16:44:31.229 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:44:31.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378633 -lt 81604378633 2026-02-25T16:44:31.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:31.454 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117000 2026-02-25T16:44:31.454 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:31.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:44:31.456 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117000 2026-02-25T16:44:31.456 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:31.457 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117000 2026-02-25T16:44:31.457 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964117000 2026-02-25T16:44:31.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117000' 2026-02-25T16:44:31.458 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:44:31.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117000 -lt 115964117000 2026-02-25T16:44:31.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:31.689 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 4-150323855367 2026-02-25T16:44:31.689 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:31.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=4 2026-02-25T16:44:31.690 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-150323855367 2026-02-25T16:44:31.690 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:31.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=150323855367 2026-02-25T16:44:31.691 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.4 seq 150323855367' 2026-02-25T16:44:31.692 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.4 seq 150323855367 2026-02-25T16:44:31.692 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-25T16:44:31.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 150323855367 -lt 150323855367 2026-02-25T16:44:31.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:31.921 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 5-184683593735 2026-02-25T16:44:31.921 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:31.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=5 2026-02-25T16:44:31.923 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-184683593735 2026-02-25T16:44:31.923 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:31.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=184683593735 2026-02-25T16:44:31.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.5 seq 184683593735' 2026-02-25T16:44:31.924 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.5 seq 184683593735 2026-02-25T16:44:31.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-25T16:44:32.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 184683593735 -lt 184683593735 2026-02-25T16:44:32.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:44:32.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:32.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:44:32.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:44:32.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T16:44:32.682 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:44:32.682 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:32.682 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:32.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T16:44:32.992 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-25T16:44:32.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-25T16:44:32.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-25T16:44:32.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T16:44:32.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-25T16:44:33.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T16:44:33.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:44:33.095 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:44:33.095 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:44:33.095 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:44:33.095 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:44:33.095 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:44:33.095 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:44:33.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T16:44:33.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:44:33.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:33.317 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:33.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-25T16:44:33.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-25T16:44:33.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-25T16:44:33.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T16:44:33.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T16:44:33.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T16:44:33.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T16:44:33.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T16:44:33.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T16:44:33.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T16:44:33.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T16:44:33.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-25T16:44:33.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-25T16:44:33.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-25T16:44:34.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-25T16:44:34.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:44:34.142 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:44:34.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:44:34.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:44:34.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:44:34.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:44:34.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:44:34.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:44:34.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:44:34.368 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:34.369 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:34.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:44:34.681 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-25T16:44:34.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:44:34.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:44:34.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:44:34.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:44:34.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:44:34.970 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:44:34.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:44:35.253 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:44:35.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:44:35.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:44:35.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:44:35.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:44:35.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:44:35.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:44:35.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:44:35.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:44:35.263 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:44:35.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:44:35.372 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:44:35.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:44:35.374 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:44:35.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:44:35.375 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:44:35.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:44:35.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:35.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:44:35.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:44:35.376 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:35.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:44:35.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:44:35.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:44:35.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:44:35.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:35.392 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:35.392 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:44:35.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:44:35.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:44:35.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:44:35.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:44:35.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:44:35.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:44:35.396 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:44:35.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:44:35.397 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:44:35.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:44:35.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:44:35.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:44:35.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:35.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:44:35.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:44:35.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:44:35.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:44:35.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:44:35.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:44:35.402 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:44:35.402 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:35.402 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:35.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:44:35.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:44:35.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:44:35.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:44:35.405 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:44:35.405 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:35.405 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:35.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:44:35.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:44:35.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:44:35.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:44:35.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:44:35.426 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:44:35.426 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:35.426 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:35.426 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:35.426 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:35.426 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:35.426 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:35.427 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:44:35.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:44:35.451 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:44:35.452 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:44:35.452 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:44:35.452 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:44:35.452 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:44:35.452 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:44:35.452 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:44:35.452 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:44:35.453 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:35.453 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:35.453 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:35.453 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:44:35.453 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:44:35.453 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:44:35.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:44:35.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:44:35.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:44:35.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:44:35.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:44:35.498 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:44:35.498 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:44:35.499 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:44:35.499 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:35.499 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:35.499 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:35.499 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:44:35.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:44:35.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:44:35.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:44:35.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:44:35.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:44:35.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:44:35.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:44:35.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:44:35.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:44:35.640 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:44:35.640 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:35.640 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:35.640 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:35.641 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:35.641 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:35.641 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:35.641 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:44:35.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:44:35.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:44:35.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:44:35.750 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:44:35.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:44:36.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:44:36.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:44:36.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:44:36.761 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:36.761 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:36.761 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:36.761 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:44:36.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:44:36.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:44:36.798 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:44:36.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:44:36.803 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:44:35.441+0000 7f626005ec00 0 load: jerasure load: lrc 2026-02-25T16:44:36.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2026-02-25T16:44:36.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2026-02-25T16:44:36.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_0: setup_osds 4 2026-02-25T16:44:36.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T16:44:36.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:44:36.805 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:44:36.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc451429-620d-42fe-9a52-38346e221c32 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dc451429-620d-42fe-9a52-38346e221c32' 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 dc451429-620d-42fe-9a52-38346e221c32 2026-02-25T16:44:36.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:36.819 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD0Jp9pZdrBMBAAkRuhMggcvdsN5I0lJy84Vg== 2026-02-25T16:44:36.819 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD0Jp9pZdrBMBAAkRuhMggcvdsN5I0lJy84Vg=="}' 2026-02-25T16:44:36.819 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc451429-620d-42fe-9a52-38346e221c32 -i td/test-erasure-eio/0/new.json 2026-02-25T16:44:36.920 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:36.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:44:36.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0Jp9pZdrBMBAAkRuhMggcvdsN5I0lJy84Vg== --osd-uuid dc451429-620d-42fe-9a52-38346e221c32 2026-02-25T16:44:36.942 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:36.941+0000 7fb56f438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:36.943 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:36.942+0000 7fb56f438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:36.944 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:36.943+0000 7fb56f438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:36.944 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:36.943+0000 7fb56f438780 -1 bdev(0x2b90c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:36.944 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:36.943+0000 7fb56f438780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:44:37.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:44:37.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:37.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:44:37.391 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:44:37.391 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:37.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:44:37.492 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:44:37.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:37.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:37.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:37.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:37.507 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:37.505+0000 7f722bc68780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:37.508 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:37.507+0000 7f722bc68780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:37.509 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:37.508+0000 7f722bc68780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:37.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:44:37.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:37.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:44:37.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:37.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:37.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:37.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:37.591 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:37.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:37.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:44:37.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:37.727 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:37.725+0000 7f722bc68780 -1 Falling back to public interface 2026-02-25T16:44:38.050 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:38.048+0000 7f722bc68780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:44:38.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:38.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:38.690 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:38.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:38.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:38.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:44:38.917 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2492447783,v1:127.0.0.1:6803/2492447783] [v2:127.0.0.1:6804/2492447783,v1:127.0.0.1:6805/2492447783] exists,up dc451429-620d-42fe-9a52-38346e221c32 2026-02-25T16:44:38.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:38.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:38.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:38.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:38.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:38.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:44:38.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:38.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:44:38.925 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:44:38.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b9cbf677-e5d2-43a0-b8e1-f526449f6692' 2026-02-25T16:44:38.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:38.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2Jp9ppNZ1NxAAG37y742SkF1HOyKMPlJC6w== 2026-02-25T16:44:38.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2Jp9ppNZ1NxAAG37y742SkF1HOyKMPlJC6w=="}' 2026-02-25T16:44:38.932 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b9cbf677-e5d2-43a0-b8e1-f526449f6692 -i td/test-erasure-eio/1/new.json 2026-02-25T16:44:39.161 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:39.168 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:44:39.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2Jp9ppNZ1NxAAG37y742SkF1HOyKMPlJC6w== --osd-uuid b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:44:39.186 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.185+0000 7f206b00e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:39.188 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.187+0000 7f206b00e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:39.189 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.187+0000 7f206b00e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:39.189 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.188+0000 7f206b00e780 -1 bdev(0x3698c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:39.189 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.188+0000 7f206b00e780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:44:39.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:44:39.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:39.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:44:39.541 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:44:39.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:39.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:44:39.865 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:44:39.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:39.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:39.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:39.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:39.879 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.878+0000 7fa37e82f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:39.881 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.879+0000 7fa37e82f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:39.881 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:39.880+0000 7fa37e82f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:40.019 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:40.017+0000 7fa37e82f780 -1 Falling back to public interface 2026-02-25T16:44:40.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:44:40.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:40.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:44:40.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:40.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:40.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:40.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:40.090 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:40.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:40.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:44:40.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:40.362 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:40.361+0000 7fa37e82f780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:44:41.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:41.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:41.315 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:41.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:41.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:41.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3379604505,v1:127.0.0.1:6811/3379604505] [v2:127.0.0.1:6812/3379604505,v1:127.0.0.1:6813/3379604505] exists,up b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:41.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:41.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:41.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:44:41.542 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:41.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ec21b9b-016b-4877-b550-f3bb50b4aeff 2026-02-25T16:44:41.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6ec21b9b-016b-4877-b550-f3bb50b4aeff' 2026-02-25T16:44:41.542 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 6ec21b9b-016b-4877-b550-f3bb50b4aeff 2026-02-25T16:44:41.543 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:41.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5Jp9pb6vfIBAAC0hZ351M8G6DcvDosqdErg== 2026-02-25T16:44:41.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5Jp9pb6vfIBAAC0hZ351M8G6DcvDosqdErg=="}' 2026-02-25T16:44:41.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ec21b9b-016b-4877-b550-f3bb50b4aeff -i td/test-erasure-eio/2/new.json 2026-02-25T16:44:41.781 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:44:41.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:44:41.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5Jp9pb6vfIBAAC0hZ351M8G6DcvDosqdErg== --osd-uuid 6ec21b9b-016b-4877-b550-f3bb50b4aeff 2026-02-25T16:44:41.806 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:41.804+0000 7f6ae300d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:41.807 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:41.806+0000 7f6ae300d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:41.808 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:41.807+0000 7f6ae300d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:41.809 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:41.807+0000 7f6ae300d780 -1 bdev(0x2dd6c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:41.809 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:41.807+0000 7f6ae300d780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:44:42.132 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:44:42.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:42.133 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:44:42.134 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:44:42.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:42.459 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:44:42.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:44:42.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:42.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:42.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:42.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:42.473 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:42.471+0000 7fd0b405f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:42.474 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:42.473+0000 7fd0b405f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:42.475 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:42.473+0000 7fd0b405f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:42.615 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:42.613+0000 7fd0b405f780 -1 Falling back to public interface 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:42.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:44:42.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:42.963 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:42.961+0000 7fd0b405f780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:44:43.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:43.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:43.910 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:43.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:43.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:43.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:44:44.137 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/949967091,v1:127.0.0.1:6819/949967091] [v2:127.0.0.1:6820/949967091,v1:127.0.0.1:6821/949967091] exists,up 6ec21b9b-016b-4877-b550-f3bb50b4aeff 2026-02-25T16:44:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:44:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:44:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:44.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:44:44.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:44.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:44:44.144 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 ce220b25-afe9-48e3-99a1-a786371fcfb0' 2026-02-25T16:44:44.145 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:44:44.145 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:44:44.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8Jp9puSzhCBAAkjGWgj9i2TjgHl3URmYJcQ== 2026-02-25T16:44:44.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8Jp9puSzhCBAAkjGWgj9i2TjgHl3URmYJcQ=="}' 2026-02-25T16:44:44.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ce220b25-afe9-48e3-99a1-a786371fcfb0 -i td/test-erasure-eio/3/new.json 2026-02-25T16:44:44.381 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:44:44.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:44:44.390 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8Jp9puSzhCBAAkjGWgj9i2TjgHl3URmYJcQ== --osd-uuid ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:44:44.406 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:44.405+0000 7f011983b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:44.408 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:44.406+0000 7f011983b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:44.409 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:44.407+0000 7f011983b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:44.409 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:44.407+0000 7f011983b780 -1 bdev(0x38bcc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:44:44.409 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:44.407+0000 7f011983b780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:44:44.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:44:44.749 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:44:44.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:44:44.750 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:44:44.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:44:45.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:44:45.068 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:44:45.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:45.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:44:45.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:44:45.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:44:45.081 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:45.079+0000 7f651a839780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:45.082 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:45.081+0000 7f651a839780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:45.083 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:45.081+0000 7f651a839780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:45.206 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:45.204+0000 7f651a839780 -1 Falling back to public interface 2026-02-25T16:44:45.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:44:45.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:45.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:44:45.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:45.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:45.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:45.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:45.295 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:44:45.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:45.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:44:45.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:45.535 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:45.533+0000 7f651a839780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:44:46.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:46.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:46.524 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:44:46.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:46.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:46.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:44:46.751 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2179351171,v1:127.0.0.1:6827/2179351171] [v2:127.0.0.1:6828/2179351171,v1:127.0.0.1:6829/2179351171] exists,up ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:44:46.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:46.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:44:46.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:44:46.798 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:44:46.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:44:46.804 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:44:37.728+0000 7f722bc68780 0 load: jerasure load: lrc 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:44:46.805 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T16:44:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:44:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:44:47.464 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:44:47.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:44:48.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:44:48.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:44:48.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:44:48.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:44:48.476 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:44:48.476 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:44:48.477 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:44:48.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:44:48.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:44:48.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:44:48.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:44:48.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:44:48.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:44:48.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:44:48.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:44:48.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:44:48.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:44:48.788 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:48.788 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:44:48.788 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:44:48.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:44:48.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:48.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:44:48.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803780 2026-02-25T16:44:48.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803780 2026-02-25T16:44:48.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780' 2026-02-25T16:44:48.852 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:48.852 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:44:48.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607556 2026-02-25T16:44:48.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607556 2026-02-25T16:44:48.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556' 2026-02-25T16:44:48.916 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:48.916 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:44:48.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411332 2026-02-25T16:44:48.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411332 2026-02-25T16:44:48.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556 2-77309411332' 2026-02-25T16:44:48.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:48.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:44:49.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149699 2026-02-25T16:44:49.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149699 2026-02-25T16:44:49.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556 2-77309411332 3-111669149699' 2026-02-25T16:44:49.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:49.045 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803780 2026-02-25T16:44:49.045 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:49.046 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:44:49.046 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803780 2026-02-25T16:44:49.046 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:49.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803780 2026-02-25T16:44:49.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803780' 2026-02-25T16:44:49.047 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803780 2026-02-25T16:44:49.048 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:49.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803778 -lt 25769803780 2026-02-25T16:44:49.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:44:50.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:44:50.277 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:50.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803780 2026-02-25T16:44:50.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:50.505 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607556 2026-02-25T16:44:50.505 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:50.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:44:50.507 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607556 2026-02-25T16:44:50.507 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:50.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607556 2026-02-25T16:44:50.508 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607556' 2026-02-25T16:44:50.508 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607556 2026-02-25T16:44:50.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:44:50.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607556 -lt 51539607556 2026-02-25T16:44:50.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:50.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411332 2026-02-25T16:44:50.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:50.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:44:50.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411332 2026-02-25T16:44:50.736 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:50.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411332 2026-02-25T16:44:50.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-25T16:44:50.737 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411332 2026-02-25T16:44:50.737 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:44:50.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411333 -lt 77309411332 2026-02-25T16:44:50.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:50.961 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149699 2026-02-25T16:44:50.962 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:50.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:44:50.963 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149699 2026-02-25T16:44:50.964 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:50.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149699 2026-02-25T16:44:50.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2026-02-25T16:44:50.965 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 111669149699 2026-02-25T16:44:50.965 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:44:51.193 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149699 -lt 111669149699 2026-02-25T16:44:51.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:44:51.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:51.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:51.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:44:51.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:44:51.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:44:51.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:44:51.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:44:51.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:44:51.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:44:51.512 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:44:51.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:44:51.731 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:44:51.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:51.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:328: TEST_rados_get_bad_size_shard_0: local shard_id=0 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:329: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T16:44:52.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-96095-0-10 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-96095-0-10 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-96095-0-10 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:44:52.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-96095-0-10 td/test-erasure-eio/ORIGINAL 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-96095-0-10 td/test-erasure-eio 0 10 set 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-0-10 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-0-10 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-0-10 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-0-10 2026-02-25T16:44:52.070 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2026-02-25T16:44:52.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:44:52.579 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:44:52.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:44:52.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T16:44:52.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T16:44:52.589 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:44:52.589 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:44:52.589 INFO:tasks.workunit.client.0.trial129.stderr:10 bytes copied, 3.6057e-05 s, 277 kB/s 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T16:44:52.590 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:44:52.591 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:44:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:44:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:44:52.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:44:52.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:44:52.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:52.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:44:52.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:44:52.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T16:44:52.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:44:52.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T16:44:52.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:53.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:44:53.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:53.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:start osd.3 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:53.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T16:44:53.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T16:44:53.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:44:53.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:44:53.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:44:53.077 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:53.075+0000 7fb529a2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:53.078 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:53.077+0000 7fb529a2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:53.079 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:53.078+0000 7fb529a2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:53.209 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:53.208+0000 7fb529a2c780 -1 Falling back to public interface 2026-02-25T16:44:53.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:53.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:44:53.516 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:53.515+0000 7fb529a2c780 -1 osd.3 35 log_to_monitors true 2026-02-25T16:44:53.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:44:54.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:44:54.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:54.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:44:54.528 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:54.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:54.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:44:54.753 INFO:tasks.workunit.client.0.trial129.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [26,35) [v2:127.0.0.1:6826/3303091837,v1:127.0.0.1:6827/3303091837] [v2:127.0.0.1:6828/3303091837,v1:127.0.0.1:6829/3303091837] exists,up ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:44:54.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:44:54.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:44:54.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:44:54.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:44:54.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:44:54.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:44:54.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:44:54.839 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:44:54.839 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:44:55.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:44:55.066 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:55.066 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:44:55.066 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:44:55.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:44:55.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:55.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:44:55.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803784 2026-02-25T16:44:55.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803784 2026-02-25T16:44:55.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784' 2026-02-25T16:44:55.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:55.131 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:44:55.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607559 2026-02-25T16:44:55.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607559 2026-02-25T16:44:55.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559' 2026-02-25T16:44:55.194 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:55.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:44:55.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411336 2026-02-25T16:44:55.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411336 2026-02-25T16:44:55.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-77309411336' 2026-02-25T16:44:55.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:44:55.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:44:55.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691843 2026-02-25T16:44:55.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691843 2026-02-25T16:44:55.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-77309411336 3-171798691843' 2026-02-25T16:44:55.321 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:55.321 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803784 2026-02-25T16:44:55.321 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:55.322 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:44:55.322 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803784 2026-02-25T16:44:55.322 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:55.323 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803784 2026-02-25T16:44:55.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-25T16:44:55.324 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803784 2026-02-25T16:44:55.324 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:55.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803784 2026-02-25T16:44:55.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:44:56.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:44:56.550 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:44:56.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-25T16:44:56.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:56.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607559 2026-02-25T16:44:56.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:56.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:44:56.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607559 2026-02-25T16:44:56.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:56.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607559 2026-02-25T16:44:56.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607559' 2026-02-25T16:44:56.783 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 51539607559 2026-02-25T16:44:56.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:44:57.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607559 -lt 51539607559 2026-02-25T16:44:57.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:57.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411336 2026-02-25T16:44:57.014 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:57.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:44:57.014 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411336 2026-02-25T16:44:57.014 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:57.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411336 2026-02-25T16:44:57.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411336' 2026-02-25T16:44:57.015 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411336 2026-02-25T16:44:57.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:44:57.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411336 -lt 77309411336 2026-02-25T16:44:57.248 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:44:57.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-171798691843 2026-02-25T16:44:57.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:44:57.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:44:57.250 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-171798691843 2026-02-25T16:44:57.250 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:44:57.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691843 2026-02-25T16:44:57.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 171798691843' 2026-02-25T16:44:57.251 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 171798691843 2026-02-25T16:44:57.252 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:44:57.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691843 -lt 171798691843 2026-02-25T16:44:57.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:44:57.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:57.478 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:57.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:44:57.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:44:57.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:44:57.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:44:57.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:44:57.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:44:57.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:44:57.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:44:58.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:44:58.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:44:58.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:44:58.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:44:58.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:44:58.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:44:58.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:44:58.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:44:58.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:44:58.647 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:44:58.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-0-10 2026-02-25T16:44:58.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:44:58.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:44:58.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-0-10 2026-02-25T16:44:58.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:44:58.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:44:58.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-0-10 td/test-erasure-eio/COPY 2026-02-25T16:44:58.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:44:58.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:44:58.677 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2026-02-25T16:44:58.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2026-02-25T16:44:58.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-96095-0-10 td/test-erasure-eio 1 10 set 2026-02-25T16:44:58.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-0-10 2026-02-25T16:44:58.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:44:58.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:44:58.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-0-10 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-0-10 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-0-10 2026-02-25T16:44:58.680 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:44:58.905 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:44:58.906 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:44:58.906 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:44:58.906 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:44:58.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:44:58.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:44:58.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T16:44:58.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:44:59.218 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:44:59.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:44:59.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T16:44:59.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T16:44:59.232 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:44:59.232 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:44:59.232 INFO:tasks.workunit.client.0.trial129.stderr:10 bytes copied, 3.4695e-05 s, 288 kB/s 2026-02-25T16:44:59.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:59.232 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:44:59.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:44:59.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:44:59.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:44:59.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:59.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:44:59.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:44:59.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:44:59.538 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:44:59.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:59.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:44:59.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:44:59.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:44:59.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:44:59.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:44:59.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:44:59.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:44:59.696 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:44:59.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:44:59.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:44:59.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:44:59.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:44:59.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:44:59.713 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:59.711+0000 7fabf4212780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:59.714 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:59.713+0000 7fabf4212780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:59.715 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:59.714+0000 7fabf4212780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:44:59.846 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:44:59.844+0000 7fabf4212780 -1 Falling back to public interface 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:44:59.923 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:44:59.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:44:59.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:00.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:00.224 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:00.222+0000 7fabf4212780 -1 osd.1 43 log_to_monitors true 2026-02-25T16:45:01.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:01.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:01.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:01.151 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:01.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:01.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:01.378 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [12,43) [v2:127.0.0.1:6810/3847623581,v1:127.0.0.1:6811/3847623581] [v2:127.0.0.1:6812/3847623581,v1:127.0.0.1:6813/3847623581] exists,up b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:45:01.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:45:01.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:45:01.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:45:01.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:45:01.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:45:01.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:45:01.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:45:01.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:45:01.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:45:01.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:45:01.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:45:01.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:45:01.690 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:01.690 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:45:01.690 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:45:01.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:45:01.690 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:01.690 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:45:01.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803787 2026-02-25T16:45:01.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803787 2026-02-25T16:45:01.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787' 2026-02-25T16:45:01.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:01.755 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:45:01.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397507 2026-02-25T16:45:01.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397507 2026-02-25T16:45:01.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-210453397507' 2026-02-25T16:45:01.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:01.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:45:01.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411339 2026-02-25T16:45:01.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411339 2026-02-25T16:45:01.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-210453397507 2-77309411339' 2026-02-25T16:45:01.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:01.882 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:45:01.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691846 2026-02-25T16:45:01.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691846 2026-02-25T16:45:01.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-210453397507 2-77309411339 3-171798691846' 2026-02-25T16:45:01.947 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:01.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803787 2026-02-25T16:45:01.948 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:01.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:45:01.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803787 2026-02-25T16:45:01.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:01.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803787 2026-02-25T16:45:01.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2026-02-25T16:45:01.950 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803787 2026-02-25T16:45:01.951 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:02.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803787 2026-02-25T16:45:02.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:03.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:45:03.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:03.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803787 2026-02-25T16:45:03.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:04.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:45:04.401 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:04.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803787 2026-02-25T16:45:04.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:04.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-210453397507 2026-02-25T16:45:04.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:04.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:45:04.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397507 2026-02-25T16:45:04.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:04.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397507 2026-02-25T16:45:04.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 210453397507' 2026-02-25T16:45:04.635 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 210453397507 2026-02-25T16:45:04.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:04.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397507 -lt 210453397507 2026-02-25T16:45:04.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:04.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411339 2026-02-25T16:45:04.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:04.864 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:45:04.865 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411339 2026-02-25T16:45:04.865 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:04.866 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411339 2026-02-25T16:45:04.867 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411339' 2026-02-25T16:45:04.867 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411339 2026-02-25T16:45:04.867 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:05.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411339 -lt 77309411339 2026-02-25T16:45:05.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:05.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-171798691846 2026-02-25T16:45:05.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:05.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:45:05.093 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-171798691846 2026-02-25T16:45:05.094 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:05.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=171798691846 2026-02-25T16:45:05.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 171798691846' 2026-02-25T16:45:05.095 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 171798691846 2026-02-25T16:45:05.096 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:05.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 171798691847 -lt 171798691846 2026-02-25T16:45:05.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:45:05.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:05.318 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:05.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:45:05.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:45:05.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:45:05.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:45:05.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:45:05.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:45:05.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:45:05.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:45:05.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:45:05.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:45:05.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:05.852 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:06.166 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:45:06.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:45:06.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:45:06.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:45:06.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:45:06.483 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:45:06.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T16:45:06.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-0-10 fail 2026-02-25T16:45:06.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:45:06.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:45:06.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-0-10 2026-02-25T16:45:06.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T16:45:06.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T16:45:06.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-96095-0-10 td/test-erasure-eio/COPY 2026-02-25T16:45:06.509 INFO:tasks.workunit.client.0.trial129.stderr:error getting pool-jerasure/obj-size-96095-0-10: (5) Input/output error 2026-02-25T16:45:06.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T16:45:06.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:45:06.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2026-02-25T16:45:06.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T16:45:06.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T16:45:06.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-96095-0-0 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-96095-0-0 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-96095-0-0 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:06.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:45:06.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:06.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:45:06.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:06.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:45:06.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:06.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:45:06.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-96095-0-0 td/test-erasure-eio/ORIGINAL 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-96095-0-0 td/test-erasure-eio 0 0 set 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-0-0 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:45:06.531 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:45:06.532 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-0-0 2026-02-25T16:45:06.532 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:45:06.532 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-0-0 2026-02-25T16:45:06.532 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-0-0 2026-02-25T16:45:06.532 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:45:06.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:45:06.757 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:06.757 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:45:06.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:45:06.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:45:06.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:45:06.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2026-02-25T16:45:06.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:45:07.070 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:45:07.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:45:07.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T16:45:07.079 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:45:07.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:07.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:07.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:07.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:07.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:45:07.388 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:45:07.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T16:45:07.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:45:07.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T16:45:07.389 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:07.549 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:07.550 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:start osd.3 2026-02-25T16:45:07.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:07.552 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T16:45:07.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T16:45:07.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:45:07.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:45:07.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:45:07.567 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:07.565+0000 7fd9b8e66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:07.568 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:07.567+0000 7fd9b8e66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:07.569 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:07.568+0000 7fd9b8e66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:07.762 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:07.761+0000 7fd9b8e66780 -1 Falling back to public interface 2026-02-25T16:45:07.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T16:45:07.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:07.777 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:45:07.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:07.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:07.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:07.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:07.778 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:45:07.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:07.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:08.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:08.105 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:08.103+0000 7fd9b8e66780 -1 osd.3 54 log_to_monitors true 2026-02-25T16:45:09.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:09.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:09.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:09.003 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:09.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:09.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:09.232 INFO:tasks.workunit.client.0.trial129.stderr:osd.3 up in weight 1 up_from 58 up_thru 58 down_at 55 last_clean_interval [40,54) [v2:127.0.0.1:6826/1271233972,v1:127.0.0.1:6827/1271233972] [v2:127.0.0.1:6828/1271233972,v1:127.0.0.1:6829/1271233972] exists,up ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:45:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:09.233 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:45:09.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:45:09.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:45:09.234 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:45:09.234 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:45:09.234 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:45:09.234 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:45:09.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:45:09.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:45:09.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:45:09.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:45:09.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:45:09.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:45:09.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:45:09.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:45:09.320 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:45:09.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:45:09.547 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:09.548 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:45:09.548 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:45:09.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:45:09.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:09.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:45:09.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803791 2026-02-25T16:45:09.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803791 2026-02-25T16:45:09.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791' 2026-02-25T16:45:09.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:09.610 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:45:09.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397510 2026-02-25T16:45:09.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397510 2026-02-25T16:45:09.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-210453397510' 2026-02-25T16:45:09.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:09.676 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:45:09.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411343 2026-02-25T16:45:09.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411343 2026-02-25T16:45:09.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-210453397510 2-77309411343' 2026-02-25T16:45:09.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:09.742 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:45:09.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103171 2026-02-25T16:45:09.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103171 2026-02-25T16:45:09.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-210453397510 2-77309411343 3-249108103171' 2026-02-25T16:45:09.806 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:09.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803791 2026-02-25T16:45:09.806 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:09.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:45:09.807 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803791 2026-02-25T16:45:09.808 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:09.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803791 2026-02-25T16:45:09.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803791' 2026-02-25T16:45:09.809 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803791 2026-02-25T16:45:09.809 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:10.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803791 -lt 25769803791 2026-02-25T16:45:10.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:10.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-210453397510 2026-02-25T16:45:10.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:10.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:45:10.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397510 2026-02-25T16:45:10.038 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:10.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=210453397510 2026-02-25T16:45:10.039 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 210453397510' 2026-02-25T16:45:10.039 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 210453397510 2026-02-25T16:45:10.039 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:10.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397508 -lt 210453397510 2026-02-25T16:45:10.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:11.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:45:11.266 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:11.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397508 -lt 210453397510 2026-02-25T16:45:11.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:12.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:45:12.494 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:12.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 210453397511 -lt 210453397510 2026-02-25T16:45:12.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:12.720 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411343 2026-02-25T16:45:12.720 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:12.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:45:12.721 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411343 2026-02-25T16:45:12.721 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:12.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411343 2026-02-25T16:45:12.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411343' 2026-02-25T16:45:12.722 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411343 2026-02-25T16:45:12.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:12.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411343 -lt 77309411343 2026-02-25T16:45:12.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:12.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-249108103171 2026-02-25T16:45:12.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:12.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:45:12.951 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-249108103171 2026-02-25T16:45:12.951 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:12.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=249108103171 2026-02-25T16:45:12.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 249108103171' 2026-02-25T16:45:12.952 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 249108103171 2026-02-25T16:45:12.952 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:13.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 249108103171 -lt 249108103171 2026-02-25T16:45:13.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:45:13.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:13.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:13.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:45:13.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:45:13.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:45:13.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:45:13.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:45:13.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:45:13.493 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:45:13.493 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:45:13.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:45:13.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:45:13.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:13.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:14.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:45:14.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:45:14.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:45:14.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:45:14.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:45:14.346 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:45:14.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-0-0 2026-02-25T16:45:14.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:45:14.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:45:14.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-0-0 2026-02-25T16:45:14.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:45:14.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:45:14.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-0-0 td/test-erasure-eio/COPY 2026-02-25T16:45:14.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:45:14.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:45:14.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-96095-0-0 td/test-erasure-eio 1 0 set 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-0-0 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:45:14.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:45:14.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-0-0 2026-02-25T16:45:14.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:45:14.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-0-0 2026-02-25T16:45:14.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-0-0 2026-02-25T16:45:14.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T16:45:14.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:45:14.950 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:45:14.960 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:45:14.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T16:45:14.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T16:45:14.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:45:14.963 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:14.964 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:14.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:14.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:14.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:15.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:15.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:15.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:45:15.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:45:15.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:45:15.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:45:15.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:45:15.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:15.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:15.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:15.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:45:15.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:45:15.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:45:15.441 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:45:15.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:15.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:45:15.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:45:15.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:45:15.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:45:15.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:45:15.451 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:15.449+0000 7f44afa0a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:15.452 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:15.451+0000 7f44afa0a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:15.453 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:15.451+0000 7f44afa0a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:15.586 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:15.585+0000 7f44afa0a780 -1 Falling back to public interface 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:15.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:15.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:15.985 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:15.983+0000 7f44afa0a780 -1 osd.1 62 log_to_monitors true 2026-02-25T16:45:16.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:16.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:16.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:16.889 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:16.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:16.889 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:17.114 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [49,62) [v2:127.0.0.1:6810/3252774334,v1:127.0.0.1:6811/3252774334] [v2:127.0.0.1:6812/3252774334,v1:127.0.0.1:6813/3252774334] exists,up b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:45:17.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:17.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:17.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:17.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:45:17.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:45:17.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:45:17.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:45:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:45:17.116 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:45:17.116 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:45:17.116 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:45:17.116 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:45:17.116 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:45:17.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:45:17.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:45:17.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:45:17.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:45:17.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:45:17.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:45:17.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:45:17.424 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:17.424 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:45:17.424 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:45:17.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:45:17.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:17.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:45:17.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803794 2026-02-25T16:45:17.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803794 2026-02-25T16:45:17.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794' 2026-02-25T16:45:17.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:17.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:45:17.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776131 2026-02-25T16:45:17.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776131 2026-02-25T16:45:17.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-292057776131' 2026-02-25T16:45:17.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:17.557 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:45:17.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411346 2026-02-25T16:45:17.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411346 2026-02-25T16:45:17.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-292057776131 2-77309411346' 2026-02-25T16:45:17.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:17.622 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:45:17.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103174 2026-02-25T16:45:17.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103174 2026-02-25T16:45:17.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-292057776131 2-77309411346 3-249108103174' 2026-02-25T16:45:17.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:17.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803794 2026-02-25T16:45:17.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:17.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:45:17.687 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803794 2026-02-25T16:45:17.687 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:17.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803794 2026-02-25T16:45:17.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803794' 2026-02-25T16:45:17.689 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803794 2026-02-25T16:45:17.689 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:17.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803794 -lt 25769803794 2026-02-25T16:45:17.923 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:17.924 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-292057776131 2026-02-25T16:45:17.924 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:17.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:45:17.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776131 2026-02-25T16:45:17.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:17.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776131 2026-02-25T16:45:17.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2026-02-25T16:45:17.926 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 292057776131 2026-02-25T16:45:17.927 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:18.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-25T16:45:18.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:18.151 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411346 2026-02-25T16:45:18.151 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:18.152 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:45:18.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411346 2026-02-25T16:45:18.153 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:18.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411346 2026-02-25T16:45:18.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411346' 2026-02-25T16:45:18.154 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411346 2026-02-25T16:45:18.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:18.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411346 -lt 77309411346 2026-02-25T16:45:18.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:18.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-249108103174 2026-02-25T16:45:18.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:18.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:45:18.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-249108103174 2026-02-25T16:45:18.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:18.381 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=249108103174 2026-02-25T16:45:18.381 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 249108103174' 2026-02-25T16:45:18.381 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 249108103174 2026-02-25T16:45:18.381 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:18.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 249108103172 -lt 249108103174 2026-02-25T16:45:18.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:19.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:45:19.609 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:19.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 249108103175 -lt 249108103174 2026-02-25T16:45:19.837 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:45:19.838 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:19.838 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:20.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:45:20.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:45:20.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:45:20.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:45:20.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:45:20.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:45:20.155 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:45:20.155 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:45:20.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:45:20.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:45:20.375 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:20.376 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:20.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:45:20.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:45:20.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:45:20.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:45:20.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:45:21.003 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:45:21.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T16:45:21.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-0-0 fail 2026-02-25T16:45:21.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:45:21.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:45:21.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-0-0 2026-02-25T16:45:21.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T16:45:21.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T16:45:21.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-96095-0-0 td/test-erasure-eio/COPY 2026-02-25T16:45:21.032 INFO:tasks.workunit.client.0.trial129.stderr:error getting pool-jerasure/obj-size-96095-0-0: (5) Input/output error 2026-02-25T16:45:21.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T16:45:21.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:45:21.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2026-02-25T16:45:21.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T16:45:21.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=0 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-96095-0-256 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-96095-0-256 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-96095-0-256 2026-02-25T16:45:21.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:45:21.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-96095-0-256 td/test-erasure-eio/ORIGINAL 2026-02-25T16:45:21.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-96095-0-256 td/test-erasure-eio 0 256 add 2026-02-25T16:45:21.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-0-256 2026-02-25T16:45:21.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:45:21.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=0 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-0-256 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-0-256 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-0-256 2026-02-25T16:45:21.053 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:45:21.281 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:45:21.281 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:21.282 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:45:21.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:45:21.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:45:21.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:45:21.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=3 2026-02-25T16:45:21.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:45:21.600 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:45:21.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T16:45:21.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:45:21.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T16:45:21.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:21.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:21.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:21.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:21.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T16:45:21.919 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:22.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T16:45:22.013 INFO:tasks.workunit.client.0.trial129.stderr:start osd.3 2026-02-25T16:45:22.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:22.014 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T16:45:22.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T16:45:22.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:45:22.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:45:22.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:45:22.027 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:22.026+0000 7f0c0861b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:22.029 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:22.027+0000 7f0c0861b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:22.030 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:22.028+0000 7f0c0861b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:22.183 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:22.181+0000 7f0c0861b780 -1 Falling back to public interface 2026-02-25T16:45:22.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T16:45:22.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:22.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:45:22.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:22.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:22.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:22.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:22.243 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:45:22.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:22.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:22.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:22.488 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:22.486+0000 7f0c0861b780 -1 osd.3 72 log_to_monitors true 2026-02-25T16:45:23.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:23.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:23.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:23.471 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:23.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:23.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:osd.3 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [58,72) [v2:127.0.0.1:6826/2072764377,v1:127.0.0.1:6827/2072764377] [v2:127.0.0.1:6828/2072764377,v1:127.0.0.1:6829/2072764377] exists,up ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:45:23.700 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:45:23.701 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:45:23.701 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:45:23.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:45:23.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:45:23.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:45:23.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:45:23.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:45:23.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:45:23.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:45:23.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:45:23.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:45:24.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:45:24.011 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:24.011 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:45:24.011 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:45:24.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:45:24.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:24.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:45:24.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803797 2026-02-25T16:45:24.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803797 2026-02-25T16:45:24.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797' 2026-02-25T16:45:24.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:24.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:45:24.139 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776134 2026-02-25T16:45:24.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776134 2026-02-25T16:45:24.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797 1-292057776134' 2026-02-25T16:45:24.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:24.140 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:45:24.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411350 2026-02-25T16:45:24.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411350 2026-02-25T16:45:24.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797 1-292057776134 2-77309411350' 2026-02-25T16:45:24.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:24.204 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:45:24.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481795 2026-02-25T16:45:24.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481795 2026-02-25T16:45:24.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797 1-292057776134 2-77309411350 3-330712481795' 2026-02-25T16:45:24.265 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:24.266 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803797 2026-02-25T16:45:24.266 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:24.267 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:45:24.267 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803797 2026-02-25T16:45:24.268 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:24.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803797 2026-02-25T16:45:24.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2026-02-25T16:45:24.269 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803797 2026-02-25T16:45:24.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:24.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803795 -lt 25769803797 2026-02-25T16:45:24.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:25.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:45:25.493 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:25.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803798 -lt 25769803797 2026-02-25T16:45:25.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:25.720 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-292057776134 2026-02-25T16:45:25.721 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:25.721 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:45:25.722 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776134 2026-02-25T16:45:25.722 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:25.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776134 2026-02-25T16:45:25.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 292057776134' 2026-02-25T16:45:25.723 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 292057776134 2026-02-25T16:45:25.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:25.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776134 -lt 292057776134 2026-02-25T16:45:25.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:25.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411350 2026-02-25T16:45:25.947 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:25.948 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:45:25.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411350 2026-02-25T16:45:25.949 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:25.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411350 2026-02-25T16:45:25.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411350' 2026-02-25T16:45:25.950 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411350 2026-02-25T16:45:25.950 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411350 -lt 77309411350 2026-02-25T16:45:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:26.175 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-330712481795 2026-02-25T16:45:26.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:26.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:45:26.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-330712481795 2026-02-25T16:45:26.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:26.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=330712481795 2026-02-25T16:45:26.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 330712481795' 2026-02-25T16:45:26.179 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 330712481795 2026-02-25T16:45:26.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:26.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 330712481795 -lt 330712481795 2026-02-25T16:45:26.404 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:45:26.404 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:26.404 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:26.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:45:26.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:45:26.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:45:26.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:45:26.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:45:26.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:45:26.719 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:45:26.719 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:45:26.938 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:45:26.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:45:26.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:26.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:27.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:45:27.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:45:27.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:45:27.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:256 bytes copied, 2.5497e-05 s, 10.0 MB/s 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:45:27.253 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:27.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-25T16:45:27.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:27.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-25T16:45:27.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:45:27.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:27.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:start osd.3 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:27.527 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-25T16:45:27.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-25T16:45:27.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:45:27.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:45:27.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:45:27.539 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:27.537+0000 7f917a93d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:27.541 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:27.539+0000 7f917a93d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:27.541 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:27.540+0000 7f917a93d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:27.732 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:27.731+0000 7f917a93d780 -1 Falling back to public interface 2026-02-25T16:45:27.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-25T16:45:27.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:27.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:45:27.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:27.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:27.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:27.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:27.753 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:45:27.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:27.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:27.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:28.084 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:28.082+0000 7f917a93d780 -1 osd.3 79 log_to_monitors true 2026-02-25T16:45:28.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:28.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:28.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:28.977 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:28.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:28.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:29.203 INFO:tasks.workunit.client.0.trial129.stderr:osd.3 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [77,79) [v2:127.0.0.1:6826/506696567,v1:127.0.0.1:6827/506696567] [v2:127.0.0.1:6828/506696567,v1:127.0.0.1:6829/506696567] exists,up ce220b25-afe9-48e3-99a1-a786371fcfb0 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:45:29.204 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:45:29.205 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:45:29.205 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:45:29.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:45:29.206 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:45:29.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:45:29.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:45:29.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:45:29.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:45:29.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:45:29.291 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:45:29.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:45:29.524 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:29.524 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:45:29.524 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:45:29.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:45:29.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:29.524 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:45:29.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803801 2026-02-25T16:45:29.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803801 2026-02-25T16:45:29.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801' 2026-02-25T16:45:29.590 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:29.590 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:45:29.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776137 2026-02-25T16:45:29.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776137 2026-02-25T16:45:29.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-292057776137' 2026-02-25T16:45:29.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:29.654 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:45:29.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411353 2026-02-25T16:45:29.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411353 2026-02-25T16:45:29.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-292057776137 2-77309411353' 2026-02-25T16:45:29.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:29.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:45:29.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285571 2026-02-25T16:45:29.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285571 2026-02-25T16:45:29.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-292057776137 2-77309411353 3-356482285571' 2026-02-25T16:45:29.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:29.782 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803801 2026-02-25T16:45:29.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:29.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:45:29.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803801 2026-02-25T16:45:29.784 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:29.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803801 2026-02-25T16:45:29.785 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2026-02-25T16:45:29.785 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803801 2026-02-25T16:45:29.785 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:30.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803801 -lt 25769803801 2026-02-25T16:45:30.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:30.017 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-292057776137 2026-02-25T16:45:30.017 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:30.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:45:30.018 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776137 2026-02-25T16:45:30.018 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:30.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776137 2026-02-25T16:45:30.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 292057776137' 2026-02-25T16:45:30.020 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 292057776137 2026-02-25T16:45:30.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:30.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776137 -lt 292057776137 2026-02-25T16:45:30.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:30.246 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411353 2026-02-25T16:45:30.246 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:30.247 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:45:30.247 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411353 2026-02-25T16:45:30.248 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:30.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411353 2026-02-25T16:45:30.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411353' 2026-02-25T16:45:30.249 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411353 2026-02-25T16:45:30.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:30.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411351 -lt 77309411353 2026-02-25T16:45:30.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:31.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:45:31.473 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:31.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411351 -lt 77309411353 2026-02-25T16:45:31.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:32.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:45:32.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:32.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411353 -lt 77309411353 2026-02-25T16:45:32.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:32.932 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-356482285571 2026-02-25T16:45:32.932 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:32.933 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:45:32.934 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-356482285571 2026-02-25T16:45:32.934 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:32.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285571 2026-02-25T16:45:32.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 356482285571' 2026-02-25T16:45:32.935 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 356482285571 2026-02-25T16:45:32.935 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:33.158 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285571 -lt 356482285571 2026-02-25T16:45:33.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:45:33.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:33.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:33.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:45:33.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:45:33.474 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:45:33.474 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:45:33.474 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:45:33.474 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:45:33.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:45:33.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:45:33.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:45:33.702 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:45:33.702 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:33.702 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:34.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:45:34.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:45:34.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:45:34.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:45:34.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:45:34.330 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:45:34.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-0-256 2026-02-25T16:45:34.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:45:34.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:45:34.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-0-256 2026-02-25T16:45:34.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:45:34.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:45:34.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-0-256 td/test-erasure-eio/COPY 2026-02-25T16:45:34.356 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:45:34.357 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:45:34.358 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 0 + 1 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=1 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-96095-0-256 td/test-erasure-eio 1 256 add 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-0-256 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T16:45:34.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:45:34.360 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-0-256 2026-02-25T16:45:34.360 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:45:34.360 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-0-256 2026-02-25T16:45:34.360 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-0-256 2026-02-25T16:45:34.360 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:45:34.588 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:45:34.588 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:34.588 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:45:34.588 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:45:34.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:45:34.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:45:34.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T16:45:34.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:45:34.934 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:34.943 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:34.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:34.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:34.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:35.251 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:35.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:35.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:45:35.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:45:35.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:45:35.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:45:35.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:45:35.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:35.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:45:35.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:45:35.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:35.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:35.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:45:35.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:45:35.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:45:35.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:45:35.351 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:45:35.360 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:35.359+0000 7fae35c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:35.362 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:35.360+0000 7fae35c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:35.362 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:35.361+0000 7fae35c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:35.515 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:35.513+0000 7fae35c32780 -1 Falling back to public interface 2026-02-25T16:45:35.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:45:35.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:35.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:45:35.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:35.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:35.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:35.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:35.574 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:45:35.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:35.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:35.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:35.862 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:35.861+0000 7fae35c32780 -1 osd.1 86 log_to_monitors true 2026-02-25T16:45:36.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:36.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:36.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:36.802 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:36.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:36.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:37.033 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 89 last_clean_interval [68,86) [v2:127.0.0.1:6810/1538851498,v1:127.0.0.1:6811/1538851498] [v2:127.0.0.1:6812/1538851498,v1:127.0.0.1:6813/1538851498] exists,up b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:45:37.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:37.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:37.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:37.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:45:37.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:45:37.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:45:37.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:45:37.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:45:37.034 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:45:37.035 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:45:37.035 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:45:37.035 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:45:37.035 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:45:37.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:45:37.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:45:37.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:45:37.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:45:37.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:45:37.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:45:37.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:45:37.347 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:37.347 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:45:37.347 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:45:37.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:45:37.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:37.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:45:37.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803804 2026-02-25T16:45:37.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803804 2026-02-25T16:45:37.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804' 2026-02-25T16:45:37.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:37.412 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:45:37.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=395136991235 2026-02-25T16:45:37.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 395136991235 2026-02-25T16:45:37.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-395136991235' 2026-02-25T16:45:37.477 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:37.477 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:45:37.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411356 2026-02-25T16:45:37.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411356 2026-02-25T16:45:37.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-395136991235 2-77309411356' 2026-02-25T16:45:37.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:37.543 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:45:37.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285574 2026-02-25T16:45:37.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285574 2026-02-25T16:45:37.607 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-395136991235 2-77309411356 3-356482285574' 2026-02-25T16:45:37.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:37.608 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803804 2026-02-25T16:45:37.608 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:37.608 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:45:37.609 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803804 2026-02-25T16:45:37.609 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:37.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803804 2026-02-25T16:45:37.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2026-02-25T16:45:37.610 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803804 2026-02-25T16:45:37.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:37.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803804 -lt 25769803804 2026-02-25T16:45:37.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:37.834 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-395136991235 2026-02-25T16:45:37.834 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:37.835 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:45:37.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-395136991235 2026-02-25T16:45:37.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:37.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=395136991235 2026-02-25T16:45:37.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 395136991235' 2026-02-25T16:45:37.837 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 395136991235 2026-02-25T16:45:37.837 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:38.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 395136991235 -lt 395136991235 2026-02-25T16:45:38.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:38.063 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411356 2026-02-25T16:45:38.063 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:38.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:45:38.064 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411356 2026-02-25T16:45:38.064 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:38.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411356 2026-02-25T16:45:38.066 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411356' 2026-02-25T16:45:38.066 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411356 2026-02-25T16:45:38.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:38.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411356 -lt 77309411356 2026-02-25T16:45:38.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:38.297 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-356482285574 2026-02-25T16:45:38.298 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:38.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:45:38.299 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-356482285574 2026-02-25T16:45:38.299 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:38.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285574 2026-02-25T16:45:38.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 356482285574' 2026-02-25T16:45:38.300 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 356482285574 2026-02-25T16:45:38.300 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:38.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285574 -lt 356482285574 2026-02-25T16:45:38.523 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:45:38.523 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:38.523 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:45:38.836 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:45:39.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:45:39.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:45:39.061 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:39.062 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:39.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:45:39.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:45:39.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:45:39.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T16:45:39.379 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:45:39.379 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:45:39.379 INFO:tasks.workunit.client.0.trial129.stderr:256 bytes copied, 2.0609e-05 s, 12.4 MB/s 2026-02-25T16:45:39.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:39.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:45:39.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:45:39.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:39.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:39.381 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:39.381 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:45:39.485 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:45:39.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:45:39.652 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:45:39.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:45:39.653 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:45:39.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:39.653 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:45:39.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:45:39.653 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:45:39.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:45:39.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:45:39.666 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:39.664+0000 7f870b410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:39.667 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:39.666+0000 7f870b410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:39.668 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:39.666+0000 7f870b410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:39.801 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:39.799+0000 7f870b410780 -1 Falling back to public interface 2026-02-25T16:45:39.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:39.880 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:40.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:40.181 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:40.179+0000 7f870b410780 -1 osd.1 93 log_to_monitors true 2026-02-25T16:45:41.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:41.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:41.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:41.110 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:41.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:41.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:41.336 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 97 up_thru 97 down_at 94 last_clean_interval [92,93) [v2:127.0.0.1:6810/3004670736,v1:127.0.0.1:6811/3004670736] [v2:127.0.0.1:6812/3004670736,v1:127.0.0.1:6813/3004670736] exists,up b9cbf677-e5d2-43a0-b8e1-f526449f6692 2026-02-25T16:45:41.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:41.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:41.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:45:41.337 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:45:41.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:45:41.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:45:41.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:45:41.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:45:41.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:45:41.424 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:45:41.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:45:41.649 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:45:41.650 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:45:41.650 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:45:41.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:45:41.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:41.650 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:45:41.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803807 2026-02-25T16:45:41.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803807 2026-02-25T16:45:41.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807' 2026-02-25T16:45:41.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:41.713 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:45:41.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=416611827715 2026-02-25T16:45:41.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 416611827715 2026-02-25T16:45:41.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807 1-416611827715' 2026-02-25T16:45:41.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:41.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:45:41.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411359 2026-02-25T16:45:41.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411359 2026-02-25T16:45:41.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807 1-416611827715 2-77309411359' 2026-02-25T16:45:41.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:45:41.844 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:45:41.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285577 2026-02-25T16:45:41.906 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285577 2026-02-25T16:45:41.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807 1-416611827715 2-77309411359 3-356482285577' 2026-02-25T16:45:41.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:41.907 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803807 2026-02-25T16:45:41.907 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:41.907 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:45:41.908 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803807 2026-02-25T16:45:41.908 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:41.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803807 2026-02-25T16:45:41.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2026-02-25T16:45:41.910 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803807 2026-02-25T16:45:41.910 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:42.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803805 -lt 25769803807 2026-02-25T16:45:42.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:43.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:45:43.135 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:43.359 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803805 -lt 25769803807 2026-02-25T16:45:43.360 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:45:44.361 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:45:44.362 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:45:44.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803807 -lt 25769803807 2026-02-25T16:45:44.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:44.590 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-416611827715 2026-02-25T16:45:44.590 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:44.591 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:45:44.591 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:44.591 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-416611827715 2026-02-25T16:45:44.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=416611827715 2026-02-25T16:45:44.592 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 416611827715' 2026-02-25T16:45:44.593 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 416611827715 2026-02-25T16:45:44.593 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:45:44.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 416611827715 -lt 416611827715 2026-02-25T16:45:44.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:44.817 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411359 2026-02-25T16:45:44.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:44.819 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:45:44.819 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411359 2026-02-25T16:45:44.819 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:44.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411359 2026-02-25T16:45:44.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411359' 2026-02-25T16:45:44.821 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 77309411359 2026-02-25T16:45:44.821 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:45:45.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411359 -lt 77309411359 2026-02-25T16:45:45.049 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:45:45.050 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-356482285577 2026-02-25T16:45:45.050 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:45:45.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:45:45.050 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-356482285577 2026-02-25T16:45:45.050 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:45:45.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=356482285577 2026-02-25T16:45:45.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 356482285577' 2026-02-25T16:45:45.051 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 356482285577 2026-02-25T16:45:45.052 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:45:45.281 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 356482285578 -lt 356482285577 2026-02-25T16:45:45.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:45:45.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:45.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:45.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:45:45.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:45:45.595 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:45:45.595 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:45:45.595 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:45:45.595 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:45:45.595 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:45:45.595 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:45:45.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:45:45.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:45:45.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:45:45.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:45:46.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:45:46.130 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:45:46.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:45:46.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:45:46.131 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:45:46.445 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:45:46.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T16:45:46.454 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-0-256 fail 2026-02-25T16:45:46.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:45:46.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:45:46.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-0-256 2026-02-25T16:45:46.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T16:45:46.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T16:45:46.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-96095-0-256 td/test-erasure-eio/COPY 2026-02-25T16:45:46.472 INFO:tasks.workunit.client.0.trial129.stderr:error getting pool-jerasure/obj-size-96095-0-256: (5) Input/output error 2026-02-25T16:45:46.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T16:45:46.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:45:46.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:332: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:45:46.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:45:46.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:45:46.710 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:45:46.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:45:46.995 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:45:47.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:45:47.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:45:47.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:45:47.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:45:47.003 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:47.003 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:47.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:47.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:47.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:47.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:47.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:45:47.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:45:47.113 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:45:47.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:45:47.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:45:47.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:45:47.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:45:47.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:45:47.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:45:47.116 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:45:47.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:45:47.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:45:47.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:45:47.133 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:45:47.133 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:47.133 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:47.134 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:45:47.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:45:47.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:45:47.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:45:47.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:45:47.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:45:47.137 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:45:47.137 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:45:47.138 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:45:47.138 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:45:47.140 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:45:47.141 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:45:47.141 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:45:47.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:45:47.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:45:47.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:45:47.142 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:45:47.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:45:47.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:45:47.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:45:47.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:45:47.144 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:47.145 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:47.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:45:47.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:45:47.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:45:47.145 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:45:47.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:45:47.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:47.147 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:47.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:45:47.148 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:45:47.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:45:47.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:45:47.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:45:47.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:45:47.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:45:47.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:45:47.149 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:45:47.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:45:47.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:45:47.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:45:47.166 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:45:47.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:47.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:47.167 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:47.167 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:47.167 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:47.167 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:47.167 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:45:47.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:45:47.191 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:45:47.191 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:45:47.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:45:47.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:45:47.192 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:45:47.193 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:45:47.194 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:45:47.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:45:47.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:45:47.235 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:45:47.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:45:47.236 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:45:47.237 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:45:47.238 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:45:47.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:45:47.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:45:47.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:45:47.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:45:47.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:45:47.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:45:47.280 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:45:47.377 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:45:47.377 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:47.377 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:47.377 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:47.377 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:47.377 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:47.377 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:47.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:45:47.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:45:47.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:45:47.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:45:47.489 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:45:47.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:45:48.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:45:48.537 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:45:48.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:45:48.543 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:45:47.182+0000 7fa6b4c27c00 0 load: jerasure load: lrc 2026-02-25T16:45:48.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2026-02-25T16:45:48.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2026-02-25T16:45:48.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:337: TEST_rados_get_bad_size_shard_1: setup_osds 4 2026-02-25T16:45:48.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T16:45:48.543 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:45:48.544 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T16:45:48.545 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T16:45:48.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:45:48.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:48.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:45:48.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:45:48.549 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:45:48.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4125ab71-84e8-4e49-aec7-22695ea0e80c 2026-02-25T16:45:48.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4125ab71-84e8-4e49-aec7-22695ea0e80c' 2026-02-25T16:45:48.550 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 4125ab71-84e8-4e49-aec7-22695ea0e80c 2026-02-25T16:45:48.550 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:45:48.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8J59px7pcIRAAQURNwU3a2Qsnth67WoNBPw== 2026-02-25T16:45:48.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8J59px7pcIRAAQURNwU3a2Qsnth67WoNBPw=="}' 2026-02-25T16:45:48.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4125ab71-84e8-4e49-aec7-22695ea0e80c -i td/test-erasure-eio/0/new.json 2026-02-25T16:45:48.663 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:45:48.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:45:48.671 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8J59px7pcIRAAQURNwU3a2Qsnth67WoNBPw== --osd-uuid 4125ab71-84e8-4e49-aec7-22695ea0e80c 2026-02-25T16:45:48.686 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:48.684+0000 7f2564c5c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:48.687 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:48.686+0000 7f2564c5c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:48.688 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:48.687+0000 7f2564c5c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:48.688 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:48.687+0000 7f2564c5c780 -1 bdev(0x2b7ac00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:45:48.688 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:48.687+0000 7f2564c5c780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:45:49.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:45:49.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:45:49.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:45:49.122 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:45:49.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:45:49.225 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:45:49.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:45:49.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:49.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:45:49.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:45:49.226 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:45:49.241 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:49.239+0000 7fb5bfa17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:49.242 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:49.240+0000 7fb5bfa17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:49.243 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:49.241+0000 7fb5bfa17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:49.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:45:49.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:49.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:45:49.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:49.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:49.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:49.320 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:45:49.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:49.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:49.320 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:45:49.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:49.464 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:49.463+0000 7fb5bfa17780 -1 Falling back to public interface 2026-02-25T16:45:49.726 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:49.724+0000 7fb5bfa17780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:45:50.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:50.418 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:50.419 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:45:50.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:50.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:50.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:45:50.643 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1899066878,v1:127.0.0.1:6803/1899066878] [v2:127.0.0.1:6804/1899066878,v1:127.0.0.1:6805/1899066878] exists,up 4125ab71-84e8-4e49-aec7-22695ea0e80c 2026-02-25T16:45:50.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:50.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:50.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:50.649 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:50.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:45:50.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:45:50.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:45:50.651 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:45:50.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:45:50.651 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a' 2026-02-25T16:45:50.651 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:45:50.652 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:45:50.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA+J59phbMjJxAAo5JOdhLOceohua9Ov4sfEQ== 2026-02-25T16:45:50.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA+J59phbMjJxAAo5JOdhLOceohua9Ov4sfEQ=="}' 2026-02-25T16:45:50.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a -i td/test-erasure-eio/1/new.json 2026-02-25T16:45:50.888 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:45:50.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:45:50.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+J59phbMjJxAAo5JOdhLOceohua9Ov4sfEQ== --osd-uuid ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:45:50.911 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:50.910+0000 7fc46e846780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:50.913 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:50.911+0000 7fc46e846780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:50.914 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:50.912+0000 7fc46e846780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:50.914 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:50.912+0000 7fc46e846780 -1 bdev(0x2668c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:45:50.914 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:50.912+0000 7fc46e846780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:45:51.259 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:45:51.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:45:51.260 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:45:51.261 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:45:51.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:45:51.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:45:51.588 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:45:51.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:51.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:45:51.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:45:51.589 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:45:51.603 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:51.601+0000 7fa7dec53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:51.604 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:51.602+0000 7fa7dec53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:51.605 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:51.603+0000 7fa7dec53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:51.753 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:51.752+0000 7fa7dec53780 -1 Falling back to public interface 2026-02-25T16:45:51.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:45:51.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:51.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:45:51.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:51.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:51.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:51.815 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:45:51.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:51.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:51.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:52.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:52.069 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:52.067+0000 7fa7dec53780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:45:53.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:53.051 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:53.051 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:45:53.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:53.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:53.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:45:53.280 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2820083790,v1:127.0.0.1:6811/2820083790] [v2:127.0.0.1:6812/2820083790,v1:127.0.0.1:6813/2820083790] exists,up ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:45:53.285 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:45:53.286 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:53.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7b57086e-5595-4d37-aa35-de88feb0c4e6' 2026-02-25T16:45:53.288 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:45:53.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:45:53.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBJ59phvZ8ERAASp0ow9wHV2ma3n7iBvCUmw== 2026-02-25T16:45:53.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBJ59phvZ8ERAASp0ow9wHV2ma3n7iBvCUmw=="}' 2026-02-25T16:45:53.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7b57086e-5595-4d37-aa35-de88feb0c4e6 -i td/test-erasure-eio/2/new.json 2026-02-25T16:45:53.526 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:45:53.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:45:53.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBJ59phvZ8ERAASp0ow9wHV2ma3n7iBvCUmw== --osd-uuid 7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:45:53.551 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:53.549+0000 7fce7786f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:53.552 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:53.551+0000 7fce7786f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:53.553 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:53.551+0000 7fce7786f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:53.553 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:53.552+0000 7fce7786f780 -1 bdev(0x3502c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:45:53.554 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:53.552+0000 7fce7786f780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:45:53.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:45:53.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:45:53.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:45:53.899 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:45:53.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:45:54.219 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:45:54.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:45:54.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:54.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:45:54.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:45:54.221 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:45:54.234 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:54.233+0000 7f831fa1b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:54.236 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:54.234+0000 7f831fa1b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:54.237 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:54.235+0000 7f831fa1b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:54.378 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:54.376+0000 7f831fa1b780 -1 Falling back to public interface 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:54.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:45:54.669 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:54.721 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:54.719+0000 7f831fa1b780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:45:55.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:55.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:55.670 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:45:55.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:55.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:55.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:45:55.894 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3913877946,v1:127.0.0.1:6819/3913877946] [v2:127.0.0.1:6820/3913877946,v1:127.0.0.1:6821/3913877946] exists,up 7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:45:55.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:55.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:55.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:55.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:45:55.898 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:45:55.899 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:45:55.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:45:55.901 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:45:55.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a5a7f056-925c-477a-b01a-6d36cd2e42f2 2026-02-25T16:45:55.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 a5a7f056-925c-477a-b01a-6d36cd2e42f2' 2026-02-25T16:45:55.902 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 a5a7f056-925c-477a-b01a-6d36cd2e42f2 2026-02-25T16:45:55.903 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:45:55.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBDJ59pol8SNhAAIiZEvqTryd98u1aaw12VRQ== 2026-02-25T16:45:55.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBDJ59pol8SNhAAIiZEvqTryd98u1aaw12VRQ=="}' 2026-02-25T16:45:55.908 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a5a7f056-925c-477a-b01a-6d36cd2e42f2 -i td/test-erasure-eio/3/new.json 2026-02-25T16:45:56.139 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:45:56.146 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:45:56.147 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDJ59pol8SNhAAIiZEvqTryd98u1aaw12VRQ== --osd-uuid a5a7f056-925c-477a-b01a-6d36cd2e42f2 2026-02-25T16:45:56.162 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.161+0000 7fc422860780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:56.164 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.162+0000 7fc422860780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:56.165 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.163+0000 7fc422860780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:56.165 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.163+0000 7fc422860780 -1 bdev(0x373ac00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:45:56.165 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.163+0000 7fc422860780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:45:56.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:45:56.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:45:56.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:45:56.516 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:45:56.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:45:56.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:45:56.833 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:45:56.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:45:56.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:45:56.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:45:56.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:45:56.847 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.846+0000 7fd25bc68780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:56.849 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.847+0000 7fd25bc68780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:56.849 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:56.848+0000 7fd25bc68780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:45:57.041 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:57.039+0000 7fd25bc68780 -1 Falling back to public interface 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:45:57.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:57.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:57.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:45:57.443 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:45:57.441+0000 7fd25bc68780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:45:58.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:45:58.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:45:58.290 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:45:58.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:45:58.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:45:58.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:45:58.518 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3524988280,v1:127.0.0.1:6827/3524988280] [v2:127.0.0.1:6828/3524988280,v1:127.0.0.1:6829/3524988280] exists,up a5a7f056-925c-477a-b01a-6d36cd2e42f2 2026-02-25T16:45:58.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:45:58.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:45:58.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:45:58.520 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:45:58.520 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:45:58.520 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:45:58.520 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:45:58.521 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:45:58.521 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:45:58.521 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:45:58.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:45:58.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:45:58.565 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:45:58.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:45:58.572 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:45:49.465+0000 7fb5bfa17780 0 load: jerasure load: lrc 2026-02-25T16:45:58.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:339: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2026-02-25T16:45:58.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T16:45:58.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:45:58.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:45:58.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T16:45:58.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:45:58.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T16:45:58.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:45:58.574 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T16:45:58.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:45:58.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:45:59.233 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:45:59.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:46:00.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:46:00.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:00.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:00.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:00.243 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:00.244 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:00.244 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:00.244 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:00.245 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:00.245 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:00.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:00.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:00.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:00.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:00.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:00.333 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:00.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:00.557 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:00.557 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:00.557 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:00.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:00.557 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:00.557 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:00.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803780 2026-02-25T16:46:00.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803780 2026-02-25T16:46:00.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780' 2026-02-25T16:46:00.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:00.625 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:00.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-25T16:46:00.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-25T16:46:00.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574852' 2026-02-25T16:46:00.689 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:00.689 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:00.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378627 2026-02-25T16:46:00.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378627 2026-02-25T16:46:00.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574852 2-81604378627' 2026-02-25T16:46:00.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:00.753 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:00.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116996 2026-02-25T16:46:00.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116996 2026-02-25T16:46:00.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574852 2-81604378627 3-115964116996' 2026-02-25T16:46:00.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:00.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803780 2026-02-25T16:46:00.818 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:00.818 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:00.819 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803780 2026-02-25T16:46:00.819 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:00.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803780 2026-02-25T16:46:00.820 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803780' 2026-02-25T16:46:00.820 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803780 2026-02-25T16:46:00.821 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:01.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803778 -lt 25769803780 2026-02-25T16:46:01.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:02.049 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:02.050 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:02.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803780 2026-02-25T16:46:02.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:02.276 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574852 2026-02-25T16:46:02.276 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:02.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:02.278 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574852 2026-02-25T16:46:02.278 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:02.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574852 2026-02-25T16:46:02.279 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574852' 2026-02-25T16:46:02.279 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574852 2026-02-25T16:46:02.280 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:02.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-25T16:46:02.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:02.504 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378627 2026-02-25T16:46:02.505 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:02.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:02.506 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378627 2026-02-25T16:46:02.506 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:02.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378627 2026-02-25T16:46:02.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378627' 2026-02-25T16:46:02.507 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378627 2026-02-25T16:46:02.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:02.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378628 -lt 81604378627 2026-02-25T16:46:02.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:02.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116996 2026-02-25T16:46:02.738 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:02.739 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:02.739 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116996 2026-02-25T16:46:02.739 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:02.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116996 2026-02-25T16:46:02.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116996' 2026-02-25T16:46:02.741 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116996 2026-02-25T16:46:02.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:02.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116996 -lt 115964116996 2026-02-25T16:46:02.966 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:02.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:02.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:03.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:03.282 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:03.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:03.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:03.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:03.282 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:03.283 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:03.283 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:03.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:03.503 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:03.503 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:03.504 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:03.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:342: TEST_rados_get_bad_size_shard_1: local shard_id=1 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:343: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T16:46:03.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=10 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-96095-1-10 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-96095-1-10 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-96095-1-10 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:46:03.834 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:03.835 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:46:03.835 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:03.835 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:46:03.835 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-96095-1-10 td/test-erasure-eio/ORIGINAL 2026-02-25T16:46:03.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-96095-1-10 td/test-erasure-eio 1 10 set 2026-02-25T16:46:03.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-1-10 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-1-10 2026-02-25T16:46:03.850 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:46:03.851 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-1-10 2026-02-25T16:46:03.851 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-1-10 2026-02-25T16:46:03.851 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:46:04.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:46:04.075 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:04.075 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:46:04.075 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:46:04.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:46:04.075 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:46:04.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T16:46:04.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:46:04.359 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:46:04.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:46:04.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T16:46:04.367 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T16:46:04.369 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:10 bytes copied, 3.6598e-05 s, 273 kB/s 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:04.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:04.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:04.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:04.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:04.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:04.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:04.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:46:04.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:04.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:04.677 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:04.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:04.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:04.847 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:04.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:04.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:46:04.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:46:04.849 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:46:04.849 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:04.849 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:46:04.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:46:04.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:04.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:04.850 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:04.858 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:04.856+0000 7fac54a39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:04.859 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:04.858+0000 7fac54a39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:04.860 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:04.859+0000 7fac54a39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:05.004 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:05.002+0000 7fac54a39780 -1 Falling back to public interface 2026-02-25T16:46:05.071 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:46:05.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:05.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:46:05.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:05.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:05.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:05.072 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:05.073 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:05.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:05.073 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:05.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:05.301 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:05.299+0000 7fac54a39780 -1 osd.1 36 log_to_monitors true 2026-02-25T16:46:06.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:06.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:06.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:06.300 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:06.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:06.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:06.525 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [13,36) [v2:127.0.0.1:6810/1530644547,v1:127.0.0.1:6811/1530644547] [v2:127.0.0.1:6812/1530644547,v1:127.0.0.1:6813/1530644547] exists,up ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:46:06.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:06.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:06.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:06.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:06.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:06.526 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:06.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:06.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:06.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:06.613 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:06.614 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:06.614 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:06.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:06.837 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:06.837 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:06.837 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:06.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:06.837 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:06.837 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:06.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803784 2026-02-25T16:46:06.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803784 2026-02-25T16:46:06.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784' 2026-02-25T16:46:06.902 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:06.903 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:06.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659139 2026-02-25T16:46:06.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659139 2026-02-25T16:46:06.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-176093659139' 2026-02-25T16:46:06.966 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:06.966 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:07.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378631 2026-02-25T16:46:07.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378631 2026-02-25T16:46:07.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-176093659139 2-81604378631' 2026-02-25T16:46:07.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:07.029 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:07.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-25T16:46:07.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-25T16:46:07.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-176093659139 2-81604378631 3-115964116999' 2026-02-25T16:46:07.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:07.096 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803784 2026-02-25T16:46:07.096 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:07.097 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:07.098 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803784 2026-02-25T16:46:07.098 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:07.099 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803784 2026-02-25T16:46:07.099 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-25T16:46:07.099 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803784 2026-02-25T16:46:07.099 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:07.324 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803784 2026-02-25T16:46:07.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:08.325 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:08.326 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:08.552 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-25T16:46:08.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:08.553 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-176093659139 2026-02-25T16:46:08.553 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:08.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:08.555 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-176093659139 2026-02-25T16:46:08.555 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:08.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=176093659139 2026-02-25T16:46:08.556 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 176093659139' 2026-02-25T16:46:08.556 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 176093659139 2026-02-25T16:46:08.556 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:08.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 176093659139 -lt 176093659139 2026-02-25T16:46:08.783 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:08.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378631 2026-02-25T16:46:08.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:08.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:08.785 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378631 2026-02-25T16:46:08.785 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:08.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378631 2026-02-25T16:46:08.786 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378631' 2026-02-25T16:46:08.786 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 81604378631 2026-02-25T16:46:08.787 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:09.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378631 -lt 81604378631 2026-02-25T16:46:09.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:09.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116999 2026-02-25T16:46:09.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:09.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:09.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116999 2026-02-25T16:46:09.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:09.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116999 2026-02-25T16:46:09.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116999' 2026-02-25T16:46:09.013 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964116999 2026-02-25T16:46:09.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:09.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-25T16:46:09.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:09.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:09.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:09.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:09.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:09.558 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:09.558 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:09.558 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:09.559 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:09.559 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:09.559 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:09.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:09.779 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:09.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:09.780 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:10.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:10.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:10.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:10.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:46:10.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:46:10.409 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:46:10.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-1-10 2026-02-25T16:46:10.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:46:10.420 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:46:10.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-1-10 2026-02-25T16:46:10.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:46:10.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:46:10.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-1-10 td/test-erasure-eio/COPY 2026-02-25T16:46:10.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:46:10.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:46:10.440 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2026-02-25T16:46:10.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2026-02-25T16:46:10.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-96095-1-10 td/test-erasure-eio 2 10 set 2026-02-25T16:46:10.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-1-10 2026-02-25T16:46:10.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=10 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-1-10 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-1-10 2026-02-25T16:46:10.443 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-1-10 2026-02-25T16:46:10.444 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2026-02-25T16:46:10.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:46:10.986 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:46:10.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:46:10.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 10 = 0 ']' 2026-02-25T16:46:10.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:10 bytes copied, 9.2563e-05 s, 108 kB/s 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T16:46:10.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:10.998 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:10.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:10.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:10.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:11.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:11.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:11.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:11.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:11.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T16:46:11.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:11.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:11.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-96095-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:11.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T16:46:11.478 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:46:11.479 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:11.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:start osd.2 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:11.481 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:46:11.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:46:11.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:11.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:11.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:11.493 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:11.491+0000 7fc040c40780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:11.494 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:11.493+0000 7fc040c40780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:11.495 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:11.494+0000 7fc040c40780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:11.650 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:11.648+0000 7fc040c40780 -1 Falling back to public interface 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:11.708 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:11.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:11.962 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:11.960+0000 7fc040c40780 -1 osd.2 45 log_to_monitors true 2026-02-25T16:46:12.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:12.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:12.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:12.940 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:12.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:12.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:13.169 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 up in weight 1 up_from 50 up_thru 38 down_at 47 last_clean_interval [19,45) [v2:127.0.0.1:6818/788783290,v1:127.0.0.1:6819/788783290] [v2:127.0.0.1:6820/788783290,v1:127.0.0.1:6821/788783290] exists,up 7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:46:13.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:13.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:13.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:13.171 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:13.171 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:13.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:13.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:13.171 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:13.256 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:13.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:13.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:13.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:13.257 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:13.257 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:13.482 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:13.483 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:13.483 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:13.483 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:13.483 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:13.484 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:13.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:13.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803787 2026-02-25T16:46:13.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803787 2026-02-25T16:46:13.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787' 2026-02-25T16:46:13.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:13.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:13.611 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659142 2026-02-25T16:46:13.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659142 2026-02-25T16:46:13.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-176093659142' 2026-02-25T16:46:13.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:13.612 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:13.675 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364803 2026-02-25T16:46:13.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364803 2026-02-25T16:46:13.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-176093659142 2-214748364803' 2026-02-25T16:46:13.676 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:13.676 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:13.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117003 2026-02-25T16:46:13.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117003 2026-02-25T16:46:13.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-176093659142 2-214748364803 3-115964117003' 2026-02-25T16:46:13.741 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:13.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803787 2026-02-25T16:46:13.741 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:13.742 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:13.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803787 2026-02-25T16:46:13.743 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:13.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803787 2026-02-25T16:46:13.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2026-02-25T16:46:13.744 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803787 2026-02-25T16:46:13.744 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:13.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803787 2026-02-25T16:46:13.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:14.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:14.974 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:15.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803787 2026-02-25T16:46:15.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:16.200 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:46:16.201 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:16.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803787 2026-02-25T16:46:16.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:16.430 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-176093659142 2026-02-25T16:46:16.430 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:16.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:16.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-176093659142 2026-02-25T16:46:16.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:16.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=176093659142 2026-02-25T16:46:16.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 176093659142' 2026-02-25T16:46:16.433 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 176093659142 2026-02-25T16:46:16.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:16.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 176093659143 -lt 176093659142 2026-02-25T16:46:16.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:16.659 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-214748364803 2026-02-25T16:46:16.659 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:16.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:16.660 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-214748364803 2026-02-25T16:46:16.660 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:16.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=214748364803 2026-02-25T16:46:16.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2026-02-25T16:46:16.662 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 214748364803 2026-02-25T16:46:16.662 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:16.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 214748364803 -lt 214748364803 2026-02-25T16:46:16.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:16.885 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117003 2026-02-25T16:46:16.885 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:16.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:16.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117003 2026-02-25T16:46:16.887 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:16.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117003 2026-02-25T16:46:16.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117003' 2026-02-25T16:46:16.888 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117003 2026-02-25T16:46:16.888 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:17.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117003 -lt 115964117003 2026-02-25T16:46:17.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:17.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:17.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:17.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:17.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:17.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:17.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:17.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:17.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:17.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:17.649 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:17.650 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:17.650 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:17.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:17.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:17.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:17.968 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:46:17.969 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:46:18.283 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:46:18.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T16:46:18.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-1-10 2026-02-25T16:46:18.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:46:18.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:46:18.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-1-10 2026-02-25T16:46:18.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:46:18.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:46:18.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-1-10 td/test-erasure-eio/COPY 2026-02-25T16:46:18.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:46:18.312 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:46:18.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:46:18.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2026-02-25T16:46:18.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T16:46:18.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=0 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=set 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-96095-1-0 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-96095-1-0 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:46:18.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-96095-1-0 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:46:18.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-96095-1-0 td/test-erasure-eio/ORIGINAL 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-96095-1-0 td/test-erasure-eio 1 0 set 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-1-0 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:46:18.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:46:18.334 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-1-0 2026-02-25T16:46:18.334 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:46:18.334 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-1-0 2026-02-25T16:46:18.334 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:46:18.334 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-1-0 2026-02-25T16:46:18.559 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:46:18.559 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:18.559 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:46:18.559 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:46:18.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:46:18.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:46:18.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T16:46:18.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:46:18.871 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:46:18.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:46:18.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T16:46:18.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T16:46:18.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:18.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:18.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:18.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:46:18.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:18.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:18.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:18.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:18.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:46:18.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:18.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:46:18.887 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:18.887 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:18.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:18.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:18.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:19.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:19.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:19.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:19.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:19.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:46:19.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:19.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:19.191 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:19.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:46:19.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:19.346 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:19.347 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:19.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:46:19.349 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:19.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:19.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:19.358 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:19.356+0000 7f6357011780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:19.360 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:19.358+0000 7f6357011780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:19.360 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:19.359+0000 7f6357011780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:19.542 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:19.540+0000 7f6357011780 -1 Falling back to public interface 2026-02-25T16:46:19.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:46:19.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:19.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:46:19.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:19.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:19.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:19.576 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:19.577 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:19.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:19.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:19.801 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:19.855 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:19.853+0000 7f6357011780 -1 osd.1 53 log_to_monitors true 2026-02-25T16:46:20.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:20.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:20.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:20.803 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:20.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:20.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:21.030 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [41,53) [v2:127.0.0.1:6810/4294962049,v1:127.0.0.1:6811/4294962049] [v2:127.0.0.1:6812/4294962049,v1:127.0.0.1:6813/4294962049] exists,up ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:46:21.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:21.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:21.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:21.030 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:21.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:21.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:21.031 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:21.031 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:21.032 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:21.032 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:21.032 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:21.032 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:21.032 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:21.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:21.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:21.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:21.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:21.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:21.120 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:21.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:21.342 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:21.342 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:21.342 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:21.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:21.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:21.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:21.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803791 2026-02-25T16:46:21.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803791 2026-02-25T16:46:21.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791' 2026-02-25T16:46:21.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:21.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:21.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=253403070467 2026-02-25T16:46:21.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 253403070467 2026-02-25T16:46:21.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-253403070467' 2026-02-25T16:46:21.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:21.472 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:21.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364806 2026-02-25T16:46:21.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364806 2026-02-25T16:46:21.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-253403070467 2-214748364806' 2026-02-25T16:46:21.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:21.534 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:21.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117006 2026-02-25T16:46:21.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117006 2026-02-25T16:46:21.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803791 1-253403070467 2-214748364806 3-115964117006' 2026-02-25T16:46:21.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:21.601 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803791 2026-02-25T16:46:21.601 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:21.601 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:21.603 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803791 2026-02-25T16:46:21.603 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:21.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803791 2026-02-25T16:46:21.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803791' 2026-02-25T16:46:21.604 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803791 2026-02-25T16:46:21.604 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:21.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803790 -lt 25769803791 2026-02-25T16:46:21.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:22.833 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:22.834 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:23.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803790 -lt 25769803791 2026-02-25T16:46:23.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:24.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:46:24.063 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:24.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803791 -lt 25769803791 2026-02-25T16:46:24.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:24.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-253403070467 2026-02-25T16:46:24.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:24.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:24.291 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-253403070467 2026-02-25T16:46:24.291 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:24.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=253403070467 2026-02-25T16:46:24.292 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 253403070467' 2026-02-25T16:46:24.292 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 253403070467 2026-02-25T16:46:24.292 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:24.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 253403070467 -lt 253403070467 2026-02-25T16:46:24.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:24.521 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-214748364806 2026-02-25T16:46:24.521 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:24.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:24.522 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-214748364806 2026-02-25T16:46:24.522 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:24.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=214748364806 2026-02-25T16:46:24.523 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 214748364806' 2026-02-25T16:46:24.523 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 214748364806 2026-02-25T16:46:24.524 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:24.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 214748364807 -lt 214748364806 2026-02-25T16:46:24.750 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:24.751 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117006 2026-02-25T16:46:24.751 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:24.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:24.752 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117006 2026-02-25T16:46:24.752 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:24.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117006 2026-02-25T16:46:24.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117006' 2026-02-25T16:46:24.754 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117006 2026-02-25T16:46:24.754 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:24.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117006 -lt 115964117006 2026-02-25T16:46:24.979 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:24.979 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:24.980 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:25.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:25.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:25.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:25.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:25.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:25.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:25.291 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:25.291 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:25.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:25.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:25.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:25.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:25.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:25.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:25.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:25.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:46:25.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:46:26.145 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:46:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-1-0 2026-02-25T16:46:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:46:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:46:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-1-0 2026-02-25T16:46:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:46:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:46:26.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-1-0 td/test-erasure-eio/COPY 2026-02-25T16:46:26.172 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:46:26.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:46:26.174 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-96095-1-0 td/test-erasure-eio 2 0 set 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-1-0 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=0 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=set 2026-02-25T16:46:26.175 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:46:26.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-1-0 2026-02-25T16:46:26.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:46:26.176 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-1-0 2026-02-25T16:46:26.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-1-0 2026-02-25T16:46:26.176 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2026-02-25T16:46:26.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:46:26.716 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:46:26.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' set = add ']' 2026-02-25T16:46:26.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:220: set_size: '[' 0 = 0 ']' 2026-02-25T16:46:26.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-25T16:46:26.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:26.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:26.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:26.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T16:46:26.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:26.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:26.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:26.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:27.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-96095-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:27.186 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:27.187 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:46:27.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:46:27.188 INFO:tasks.workunit.client.0.trial129.stderr:start osd.2 2026-02-25T16:46:27.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:27.188 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:46:27.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:46:27.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:27.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:27.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:27.202 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:27.201+0000 7f9cb6b3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:27.204 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:27.202+0000 7f9cb6b3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:27.205 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:27.203+0000 7f9cb6b3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:27.336 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:27.335+0000 7f9cb6b3d780 -1 Falling back to public interface 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:27.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:27.640 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:27.661 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:27.659+0000 7f9cb6b3d780 -1 osd.2 63 log_to_monitors true 2026-02-25T16:46:28.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:28.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:28.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:28.642 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:28.642 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:28.643 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 up in weight 1 up_from 68 up_thru 56 down_at 65 last_clean_interval [50,63) [v2:127.0.0.1:6818/2321630712,v1:127.0.0.1:6819/2321630712] [v2:127.0.0.1:6820/2321630712,v1:127.0.0.1:6821/2321630712] exists,up 7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:28.865 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:28.866 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:28.866 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:28.866 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:28.866 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:28.866 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:28.866 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:28.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:28.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:28.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:28.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:28.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:28.952 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:29.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:29.177 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:29.177 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:29.177 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:29.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:29.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:29.177 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:29.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803794 2026-02-25T16:46:29.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803794 2026-02-25T16:46:29.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794' 2026-02-25T16:46:29.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:29.241 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:29.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=253403070470 2026-02-25T16:46:29.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 253403070470 2026-02-25T16:46:29.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-253403070470' 2026-02-25T16:46:29.307 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:29.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:29.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776131 2026-02-25T16:46:29.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776131 2026-02-25T16:46:29.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-253403070470 2-292057776131' 2026-02-25T16:46:29.371 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:29.371 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:29.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117010 2026-02-25T16:46:29.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117010 2026-02-25T16:46:29.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803794 1-253403070470 2-292057776131 3-115964117010' 2026-02-25T16:46:29.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:29.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803794 2026-02-25T16:46:29.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:29.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:29.437 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803794 2026-02-25T16:46:29.437 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:29.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803794 2026-02-25T16:46:29.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803794' 2026-02-25T16:46:29.438 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803794 2026-02-25T16:46:29.439 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:29.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803794 -lt 25769803794 2026-02-25T16:46:29.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:29.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-253403070470 2026-02-25T16:46:29.661 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:29.662 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:29.663 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-253403070470 2026-02-25T16:46:29.663 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:29.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=253403070470 2026-02-25T16:46:29.664 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 253403070470' 2026-02-25T16:46:29.664 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 253403070470 2026-02-25T16:46:29.665 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:29.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 253403070470 -lt 253403070470 2026-02-25T16:46:29.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:29.893 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-292057776131 2026-02-25T16:46:29.893 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:29.893 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:29.894 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776131 2026-02-25T16:46:29.894 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:29.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776131 2026-02-25T16:46:29.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 292057776131' 2026-02-25T16:46:29.895 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 292057776131 2026-02-25T16:46:29.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:30.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-25T16:46:30.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:30.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117010 2026-02-25T16:46:30.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:30.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:30.122 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117010 2026-02-25T16:46:30.122 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:30.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117010 2026-02-25T16:46:30.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117010' 2026-02-25T16:46:30.124 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117010 2026-02-25T16:46:30.124 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:30.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117008 -lt 115964117010 2026-02-25T16:46:30.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:31.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:31.354 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:31.579 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117010 -lt 115964117010 2026-02-25T16:46:31.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:31.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:31.580 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:31.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:31.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:31.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:31.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:31.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:31.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:31.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:31.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:32.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:32.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:32.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:32.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:32.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:32.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:32.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:32.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:46:32.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:46:32.748 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:46:32.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T16:46:32.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-1-0 2026-02-25T16:46:32.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:46:32.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:46:32.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-1-0 2026-02-25T16:46:32.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:46:32.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:46:32.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-1-0 td/test-erasure-eio/COPY 2026-02-25T16:46:32.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:46:32.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:46:32.776 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:233: rados_get_data_bad_size: shift 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:234: rados_get_data_bad_size: local shard_id=1 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: shift 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: local bytes=256 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: shift 2026-02-25T16:46:32.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: local mode=add 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: local objname=obj-size-96095-1-256 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-96095-1-256 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-96095-1-256 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:46:32.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:46:32.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:46:32.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-96095-1-256 td/test-erasure-eio/ORIGINAL 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:246: rados_get_data_bad_size: set_size obj-size-96095-1-256 td/test-erasure-eio 1 256 add 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-1-256 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=1 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-1-256 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-1-256 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-1-256 2026-02-25T16:46:32.795 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:46:33.026 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:46:33.027 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:33.027 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:46:33.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:46:33.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:46:33.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:46:33.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=1 2026-02-25T16:46:33.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:46:33.310 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:46:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T16:46:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:33.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:33.319 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:33.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:33.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:33.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:33.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:33.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:46:33.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:33.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:33.627 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:33.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:46:33.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:33.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:33.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:46:33.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:33.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:33.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:33.735 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:33.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:33.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:33.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:46:33.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:46:33.736 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:46:33.736 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:33.737 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:46:33.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:46:33.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:33.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:33.737 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:33.747 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:33.745+0000 7f8ef73d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:33.748 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:33.747+0000 7f8ef73d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:33.749 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:33.747+0000 7f8ef73d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:33.880 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:33.879+0000 7f8ef73d4780 -1 Falling back to public interface 2026-02-25T16:46:33.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:46:33.961 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:33.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:34.198 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:34.196+0000 7f8ef73d4780 -1 osd.1 73 log_to_monitors true 2026-02-25T16:46:34.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:35.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:35.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:35.204 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:35.204 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:35.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:35.205 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:35.431 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [59,73) [v2:127.0.0.1:6810/3865785483,v1:127.0.0.1:6811/3865785483] [v2:127.0.0.1:6812/3865785483,v1:127.0.0.1:6813/3865785483] exists,up ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:46:35.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:35.432 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:35.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:35.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:35.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:35.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:35.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:35.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:35.521 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:35.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:35.746 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:35.746 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:35.747 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:35.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:35.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:35.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:35.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803797 2026-02-25T16:46:35.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803797 2026-02-25T16:46:35.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797' 2026-02-25T16:46:35.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:35.809 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:35.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481795 2026-02-25T16:46:35.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481795 2026-02-25T16:46:35.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797 1-330712481795' 2026-02-25T16:46:35.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:35.873 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:35.939 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776134 2026-02-25T16:46:35.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776134 2026-02-25T16:46:35.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797 1-330712481795 2-292057776134' 2026-02-25T16:46:35.940 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:35.940 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:36.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117013 2026-02-25T16:46:36.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117013 2026-02-25T16:46:36.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803797 1-330712481795 2-292057776134 3-115964117013' 2026-02-25T16:46:36.004 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:36.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803797 2026-02-25T16:46:36.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:36.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:36.006 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803797 2026-02-25T16:46:36.006 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:36.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803797 2026-02-25T16:46:36.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2026-02-25T16:46:36.008 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803797 2026-02-25T16:46:36.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:36.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803795 -lt 25769803797 2026-02-25T16:46:36.236 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:37.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:37.238 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:37.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803798 -lt 25769803797 2026-02-25T16:46:37.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:37.466 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-330712481795 2026-02-25T16:46:37.466 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:37.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:37.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481795 2026-02-25T16:46:37.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:37.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=330712481795 2026-02-25T16:46:37.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 330712481795' 2026-02-25T16:46:37.469 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 330712481795 2026-02-25T16:46:37.469 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:37.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 330712481795 -lt 330712481795 2026-02-25T16:46:37.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:37.700 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-292057776134 2026-02-25T16:46:37.700 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:37.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:37.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776134 2026-02-25T16:46:37.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:37.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776134 2026-02-25T16:46:37.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 292057776134' 2026-02-25T16:46:37.702 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 292057776134 2026-02-25T16:46:37.702 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:37.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776134 -lt 292057776134 2026-02-25T16:46:37.924 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:37.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117013 2026-02-25T16:46:37.925 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:37.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:37.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117013 2026-02-25T16:46:37.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:37.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117013 2026-02-25T16:46:37.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117013' 2026-02-25T16:46:37.927 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117013 2026-02-25T16:46:37.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:38.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117013 -lt 115964117013 2026-02-25T16:46:38.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:38.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:38.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:38.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:38.468 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:38.469 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:38.469 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:38.469 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:38.469 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:38.469 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:38.469 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:38.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:38.693 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:38.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:38.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:39.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:39.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:39.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:39.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T16:46:39.007 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:256 bytes copied, 2.1601e-05 s, 11.9 MB/s 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:46:39.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:39.009 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:39.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:39.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:39.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:39.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:39.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:39.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:39.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:39.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:46:39.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:39.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:39.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:39.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:39.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:46:39.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:46:39.272 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:46:39.273 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:39.273 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:46:39.274 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:46:39.274 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:39.274 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:39.274 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:39.287 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:39.285+0000 7fd22fa2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:39.288 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:39.286+0000 7fd22fa2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:39.288 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:39.287+0000 7fd22fa2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:39.418 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:39.416+0000 7fd22fa2d780 -1 Falling back to public interface 2026-02-25T16:46:39.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:39.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:39.722 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:39.720+0000 7fd22fa2d780 -1 osd.1 78 log_to_monitors true 2026-02-25T16:46:39.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:40.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:40.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:40.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:40.731 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:40.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:40.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:46:40.964 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [77,78) [v2:127.0.0.1:6810/882127350,v1:127.0.0.1:6811/882127350] [v2:127.0.0.1:6812/882127350,v1:127.0.0.1:6813/882127350] exists,up ff62ab01-1ccb-444f-ae3c-38f64a9a6e3a 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:40.965 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:40.966 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:40.966 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:40.966 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:40.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:40.967 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:41.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:41.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:41.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:41.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:41.053 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:41.053 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:41.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:41.275 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:41.275 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:41.275 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:41.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:41.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:41.276 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:41.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803801 2026-02-25T16:46:41.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803801 2026-02-25T16:46:41.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801' 2026-02-25T16:46:41.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:41.340 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:41.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318275 2026-02-25T16:46:41.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318275 2026-02-25T16:46:41.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-352187318275' 2026-02-25T16:46:41.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:41.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:41.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776137 2026-02-25T16:46:41.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776137 2026-02-25T16:46:41.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-352187318275 2-292057776137' 2026-02-25T16:46:41.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:41.470 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:41.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117016 2026-02-25T16:46:41.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117016 2026-02-25T16:46:41.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803801 1-352187318275 2-292057776137 3-115964117016' 2026-02-25T16:46:41.533 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:41.533 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803801 2026-02-25T16:46:41.533 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:41.534 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:41.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803801 2026-02-25T16:46:41.535 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:41.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803801 2026-02-25T16:46:41.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2026-02-25T16:46:41.537 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803801 2026-02-25T16:46:41.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:41.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803801 -lt 25769803801 2026-02-25T16:46:41.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:41.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-352187318275 2026-02-25T16:46:41.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:41.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:41.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-352187318275 2026-02-25T16:46:41.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:41.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=352187318275 2026-02-25T16:46:41.764 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2026-02-25T16:46:41.764 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 352187318275 2026-02-25T16:46:41.764 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:41.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 352187318274 -lt 352187318275 2026-02-25T16:46:41.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:42.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:42.990 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:43.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 352187318274 -lt 352187318275 2026-02-25T16:46:43.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:44.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:46:44.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:44.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 352187318275 -lt 352187318275 2026-02-25T16:46:44.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:44.442 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-292057776137 2026-02-25T16:46:44.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:44.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:44.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776137 2026-02-25T16:46:44.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:44.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=292057776137 2026-02-25T16:46:44.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 292057776137' 2026-02-25T16:46:44.445 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 292057776137 2026-02-25T16:46:44.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:44.668 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 292057776138 -lt 292057776137 2026-02-25T16:46:44.668 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:44.669 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:44.669 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117016 2026-02-25T16:46:44.669 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:44.670 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117016 2026-02-25T16:46:44.670 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:44.671 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117016 2026-02-25T16:46:44.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117016' 2026-02-25T16:46:44.672 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117016 2026-02-25T16:46:44.672 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:44.897 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117016 -lt 115964117016 2026-02-25T16:46:44.898 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:44.898 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:44.898 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:45.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:45.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:45.440 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:45.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:45.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:45.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:45.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:45.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:45.752 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:46:45.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:46:46.068 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:46:46.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:248: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-1-256 2026-02-25T16:46:46.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:46:46.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:46:46.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-1-256 2026-02-25T16:46:46.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:46:46.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:46:46.078 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-1-256 td/test-erasure-eio/COPY 2026-02-25T16:46:46.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:46:46.096 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:46:46.097 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: expr 1 + 1 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:251: rados_get_data_bad_size: local another_shard_id=2 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:252: rados_get_data_bad_size: set_size obj-size-96095-1-256 td/test-erasure-eio 2 256 add 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local objname=obj-size-96095-1-256 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: shift 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local dir=td/test-erasure-eio 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: shift 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: local shard_id=2 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: shift 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: local bytes=256 2026-02-25T16:46:46.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: shift 2026-02-25T16:46:46.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: local mode=add 2026-02-25T16:46:46.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: local poolname=pool-jerasure 2026-02-25T16:46:46.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: get_osds pool-jerasure obj-size-96095-1-256 2026-02-25T16:46:46.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:46:46.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-size-96095-1-256 2026-02-25T16:46:46.101 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-size-96095-1-256 2026-02-25T16:46:46.101 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: initial_osds=('3' '1' '2') 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: local -a initial_osds 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: local osd_id=2 2026-02-25T16:46:46.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: ceph osd set noout 2026-02-25T16:46:46.674 INFO:tasks.workunit.client.0.trial129.stderr:noout is set 2026-02-25T16:46:46.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: '[' add = add ']' 2026-02-25T16:46:46.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:46.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:46.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:46.685 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:46.686 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:46.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:46.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-96095-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:47.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T16:46:47.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:47.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:47.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:47.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:47.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:start osd.2 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:47.093 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:46:47.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:46:47.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:47.094 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:47.095 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:47.104 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:47.102+0000 7f7a10848780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:47.105 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:47.103+0000 7f7a10848780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:47.106 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:47.104+0000 7f7a10848780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:47.287 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:47.286+0000 7f7a10848780 -1 Falling back to public interface 2026-02-25T16:46:47.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:46:47.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:47.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:46:47.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:47.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:47.314 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:47.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:47.315 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:47.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:47.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:47.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:47.667 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:47.665+0000 7f7a10848780 -1 osd.2 85 log_to_monitors true 2026-02-25T16:46:48.541 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:48.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:48.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:48.542 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:48.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:48.542 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:48.768 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 up in weight 1 up_from 91 up_thru 79 down_at 88 last_clean_interval [68,85) [v2:127.0.0.1:6818/582959108,v1:127.0.0.1:6819/582959108] [v2:127.0.0.1:6820/582959108,v1:127.0.0.1:6821/582959108] exists,up 7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:46:48.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:48.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:48.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:48.768 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:48.769 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:48.769 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:48.769 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:48.769 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:48.769 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:48.769 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:48.770 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:48.770 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:48.770 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:48.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:48.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:48.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:48.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:48.857 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:48.857 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:49.084 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:49.084 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:49.085 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:49.085 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:49.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:49.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:49.085 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:49.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803804 2026-02-25T16:46:49.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803804 2026-02-25T16:46:49.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804' 2026-02-25T16:46:49.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:49.150 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:49.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318278 2026-02-25T16:46:49.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318278 2026-02-25T16:46:49.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-352187318278' 2026-02-25T16:46:49.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:49.214 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:49.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023939 2026-02-25T16:46:49.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023939 2026-02-25T16:46:49.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-352187318278 2-390842023939' 2026-02-25T16:46:49.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:49.277 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:49.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117020 2026-02-25T16:46:49.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117020 2026-02-25T16:46:49.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803804 1-352187318278 2-390842023939 3-115964117020' 2026-02-25T16:46:49.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:49.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803804 2026-02-25T16:46:49.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:49.342 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:49.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803804 2026-02-25T16:46:49.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:49.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803804 2026-02-25T16:46:49.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2026-02-25T16:46:49.344 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803804 2026-02-25T16:46:49.344 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:49.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803804 -lt 25769803804 2026-02-25T16:46:49.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:49.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-352187318278 2026-02-25T16:46:49.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:49.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:49.571 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-352187318278 2026-02-25T16:46:49.572 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:49.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=352187318278 2026-02-25T16:46:49.573 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 352187318278' 2026-02-25T16:46:49.573 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 352187318278 2026-02-25T16:46:49.573 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:49.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 352187318278 -lt 352187318278 2026-02-25T16:46:49.800 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:49.801 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-390842023939 2026-02-25T16:46:49.801 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:49.802 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:49.803 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-390842023939 2026-02-25T16:46:49.803 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:49.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=390842023939 2026-02-25T16:46:49.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 390842023939' 2026-02-25T16:46:49.804 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 390842023939 2026-02-25T16:46:49.804 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:50.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 390842023939 -lt 390842023939 2026-02-25T16:46:50.033 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:50.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117020 2026-02-25T16:46:50.034 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:50.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:50.035 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117020 2026-02-25T16:46:50.035 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:50.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117020 2026-02-25T16:46:50.037 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117020' 2026-02-25T16:46:50.037 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117020 2026-02-25T16:46:50.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:50.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117020 -lt 115964117020 2026-02-25T16:46:50.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:50.264 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:50.264 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:50.581 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:50.581 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:50.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:50.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:50.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:50.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:50.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:50.582 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:50.807 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:50.807 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:50.807 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:50.807 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:51.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:51.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:51.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:51.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:219: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records in 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:1+0 records out 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:256 bytes copied, 2.124e-05 s, 12.1 MB/s 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:51.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:51.126 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:51.229 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:51.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:51.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:46:51.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:46:51.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T16:46:51.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:46:51.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:51.230 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-96095-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-25T16:46:51.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T16:46:51.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:46:51.396 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:46:51.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:46:51.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:start osd.2 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:46:51.399 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:46:51.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:46:51.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:46:51.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:46:51.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:46:51.410 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:51.408+0000 7f0aa7217780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:51.411 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:51.410+0000 7f0aa7217780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:51.412 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:51.410+0000 7f0aa7217780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:46:51.553 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:51.551+0000 7f0aa7217780 -1 Falling back to public interface 2026-02-25T16:46:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:46:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:46:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:46:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:46:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:46:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:51.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:46:51.630 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:46:51.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:51.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:51.851 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:46:51.923 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:46:51.922+0000 7f0aa7217780 -1 osd.2 92 log_to_monitors true 2026-02-25T16:46:52.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:46:52.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:46:52.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:46:52.854 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:52.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:46:52.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:46:53.080 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 up in weight 1 up_from 96 up_thru 79 down_at 93 last_clean_interval [91,92) [v2:127.0.0.1:6818/1748358885,v1:127.0.0.1:6819/1748358885] [v2:127.0.0.1:6820/1748358885,v1:127.0.0.1:6821/1748358885] exists,up 7b57086e-5595-4d37-aa35-de88feb0c4e6 2026-02-25T16:46:53.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:46:53.081 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:46:53.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:46:53.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:46:53.082 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:46:53.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:46:53.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:46:53.169 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:46:53.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:46:53.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:46:53.170 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:46:53.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:46:53.397 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:46:53.397 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:46:53.397 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:46:53.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:46:53.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:53.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:46:53.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803807 2026-02-25T16:46:53.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803807 2026-02-25T16:46:53.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807' 2026-02-25T16:46:53.461 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:53.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:46:53.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318281 2026-02-25T16:46:53.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318281 2026-02-25T16:46:53.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807 1-352187318281' 2026-02-25T16:46:53.525 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:53.525 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:46:53.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860419 2026-02-25T16:46:53.587 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860419 2026-02-25T16:46:53.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807 1-352187318281 2-412316860419' 2026-02-25T16:46:53.588 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:46:53.588 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:46:53.654 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117022 2026-02-25T16:46:53.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117022 2026-02-25T16:46:53.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803807 1-352187318281 2-412316860419 3-115964117022' 2026-02-25T16:46:53.655 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:53.655 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803807 2026-02-25T16:46:53.655 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:53.656 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:46:53.656 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803807 2026-02-25T16:46:53.656 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:53.657 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803807 2026-02-25T16:46:53.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2026-02-25T16:46:53.658 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803807 2026-02-25T16:46:53.658 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:53.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803805 -lt 25769803807 2026-02-25T16:46:53.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:54.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:46:54.884 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:55.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803805 -lt 25769803807 2026-02-25T16:46:55.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:46:56.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:46:56.111 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:46:56.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803807 -lt 25769803807 2026-02-25T16:46:56.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:56.336 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-352187318281 2026-02-25T16:46:56.336 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:56.337 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:46:56.337 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-352187318281 2026-02-25T16:46:56.338 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:56.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=352187318281 2026-02-25T16:46:56.339 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 352187318281' 2026-02-25T16:46:56.339 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 352187318281 2026-02-25T16:46:56.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:46:56.563 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 352187318282 -lt 352187318281 2026-02-25T16:46:56.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:56.564 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-412316860419 2026-02-25T16:46:56.564 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:56.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:46:56.566 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-412316860419 2026-02-25T16:46:56.566 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:56.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=412316860419 2026-02-25T16:46:56.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 412316860419' 2026-02-25T16:46:56.567 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 412316860419 2026-02-25T16:46:56.567 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:46:56.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 412316860419 -lt 412316860419 2026-02-25T16:46:56.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:46:56.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117022 2026-02-25T16:46:56.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:46:56.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:46:56.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117022 2026-02-25T16:46:56.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:46:56.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117022 2026-02-25T16:46:56.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117022' 2026-02-25T16:46:56.797 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117022 2026-02-25T16:46:56.797 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:46:57.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117023 -lt 115964117022 2026-02-25T16:46:57.024 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:46:57.025 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:57.025 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:57.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:46:57.338 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:46:57.338 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:46:57.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:46:57.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:46:57.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:46:57.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:46:57.339 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:46:57.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:46:57.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:46:57.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:46:57.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:46:57.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:46:57.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:46:57.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:46:57.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-25T16:46:57.873 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: set_size: ceph osd unset noout 2026-02-25T16:46:58.187 INFO:tasks.workunit.client.0.trial129.stderr:noout is unset 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:253: rados_get_data_bad_size: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-96095-1-256 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-96095-1-256 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:46:58.196 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-96095-1-256 td/test-erasure-eio/COPY 2026-02-25T16:46:58.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:46:58.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:46:58.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:264: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:46:58.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:346: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:46:58.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:46:58.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:46:58.451 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:46:58.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:46:58.738 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:46:58.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:46:58.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:46:58.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:46:58.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:46:58.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:58.747 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:58.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:58.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:58.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:58.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:58.856 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:46:58.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:46:58.857 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:46:58.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:46:58.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:46:58.858 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:46:58.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:46:58.859 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:46:58.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:46:58.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:46:58.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:46:58.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:46:58.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:46:58.879 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:46:58.879 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:58.879 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:58.879 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:46:58.881 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:46:58.882 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:46:58.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:46:58.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:46:58.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:46:58.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:46:58.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:46:58.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:46:58.884 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:46:58.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:46:58.885 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:46:58.885 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:46:58.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:46:58.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:46:58.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:46:58.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:46:58.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:46:58.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:46:58.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:46:58.889 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:46:58.889 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:58.890 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:58.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:46:58.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:46:58.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:46:58.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:46:58.892 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:46:58.892 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:58.892 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:58.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:46:58.893 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:46:58.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:46:58.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:46:58.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:46:58.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:46:58.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:46:58.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:46:58.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:46:58.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:46:58.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:46:58.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:46:58.913 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:46:58.913 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:58.914 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:58.914 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:58.914 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:58.914 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:58.914 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:58.914 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:46:58.937 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:46:58.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:46:58.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:46:58.938 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:46:58.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:46:58.939 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:46:58.940 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:46:58.941 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:46:58.983 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:46:58.984 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:46:58.985 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:58.985 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:58.985 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:46:58.985 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:46:58.985 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:46:59.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:46:59.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:46:59.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:46:59.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:46:59.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:46:59.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:46:59.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:46:59.125 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:46:59.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:46:59.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:46:59.126 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:46:59.126 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:46:59.126 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:46:59.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:46:59.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:46:59.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:46:59.142 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:46:59.143 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:46:59.234 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:46:59.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:47:00.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:47:00.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:47:00.282 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:47:00.286 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:47:00.287 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:46:58.929+0000 7f8680042c00 0 load: jerasure load: lrc 2026-02-25T16:47:00.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2026-02-25T16:47:00.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2026-02-25T16:47:00.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:276: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2026-02-25T16:47:00.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T16:47:00.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:47:00.288 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T16:47:00.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:00.294 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:00.295 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:00.296 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:00.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:00.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:47:00.297 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:00.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=59861638-330c-40fd-b52a-c0fd72966e19 2026-02-25T16:47:00.297 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 59861638-330c-40fd-b52a-c0fd72966e19' 2026-02-25T16:47:00.297 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 59861638-330c-40fd-b52a-c0fd72966e19 2026-02-25T16:47:00.298 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:00.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCEJ59p1pMDEhAAjOSOamQuqTNwz0gGBWS7GA== 2026-02-25T16:47:00.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCEJ59p1pMDEhAAjOSOamQuqTNwz0gGBWS7GA=="}' 2026-02-25T16:47:00.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 59861638-330c-40fd-b52a-c0fd72966e19 -i td/test-erasure-eio/0/new.json 2026-02-25T16:47:00.406 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:00.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:47:00.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEJ59p1pMDEhAAjOSOamQuqTNwz0gGBWS7GA== --osd-uuid 59861638-330c-40fd-b52a-c0fd72966e19 2026-02-25T16:47:00.430 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.428+0000 7f1d0461a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:00.431 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.429+0000 7f1d0461a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:00.432 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.430+0000 7f1d0461a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:00.432 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.431+0000 7f1d0461a780 -1 bdev(0x36f2c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:00.432 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.431+0000 7f1d0461a780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:47:00.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:47:00.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:00.877 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:47:00.877 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:47:00.877 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:00.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:47:00.979 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:47:00.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:00.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:00.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:00.980 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:00.994 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.992+0000 7fb5b6a34780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:00.995 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.993+0000 7fb5b6a34780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:00.996 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:00.994+0000 7fb5b6a34780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:01.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:47:01.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:01.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:47:01.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:01.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:01.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:01.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:01.077 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:01.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:01.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:47:01.176 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:01.174+0000 7fb5b6a34780 -1 Falling back to public interface 2026-02-25T16:47:01.177 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:01.450 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:01.448+0000 7fb5b6a34780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:47:02.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:02.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:02.179 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:02.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:02.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:02.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:47:02.405 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2184612389,v1:127.0.0.1:6803/2184612389] [v2:127.0.0.1:6804/2184612389,v1:127.0.0.1:6805/2184612389] exists,up 59861638-330c-40fd-b52a-c0fd72966e19 2026-02-25T16:47:02.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:02.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:02.411 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:47:02.413 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e4739c2d-320f-4276-9a86-9e051c4ae95c 2026-02-25T16:47:02.413 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e4739c2d-320f-4276-9a86-9e051c4ae95c' 2026-02-25T16:47:02.413 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 e4739c2d-320f-4276-9a86-9e051c4ae95c 2026-02-25T16:47:02.414 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:02.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGJ59p3I7mGBAAQEs3uQKGwIxn4DVekQsdJg== 2026-02-25T16:47:02.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGJ59p3I7mGBAAQEs3uQKGwIxn4DVekQsdJg=="}' 2026-02-25T16:47:02.419 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e4739c2d-320f-4276-9a86-9e051c4ae95c -i td/test-erasure-eio/1/new.json 2026-02-25T16:47:02.650 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:02.658 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:47:02.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGJ59p3I7mGBAAQEs3uQKGwIxn4DVekQsdJg== --osd-uuid e4739c2d-320f-4276-9a86-9e051c4ae95c 2026-02-25T16:47:02.673 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:02.671+0000 7f5590433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:02.675 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:02.673+0000 7f5590433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:02.675 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:02.674+0000 7f5590433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:02.676 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:02.674+0000 7f5590433780 -1 bdev(0x3bf6c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:02.676 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:02.674+0000 7f5590433780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:47:03.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:47:03.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:03.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:47:03.060 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:47:03.060 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:03.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:47:03.378 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:47:03.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:03.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:03.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:03.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:03.392 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:03.391+0000 7ffbbc072780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:03.394 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:03.392+0000 7ffbbc072780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:03.394 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:03.393+0000 7ffbbc072780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:03.601 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:03.600+0000 7ffbbc072780 -1 Falling back to public interface 2026-02-25T16:47:03.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:47:03.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:03.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:47:03.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:03.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:03.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:03.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:03.616 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:03.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:03.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:03.840 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:03.969 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:03.967+0000 7ffbbc072780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:47:04.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:04.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:04.841 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:04.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:04.841 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:04.842 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:05.067 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1377920152,v1:127.0.0.1:6811/1377920152] [v2:127.0.0.1:6812/1377920152,v1:127.0.0.1:6813/1377920152] exists,up e4739c2d-320f-4276-9a86-9e051c4ae95c 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:05.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:05.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:47:05.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:05.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=861cd02e-ed7a-400d-9db9-66582946fb54 2026-02-25T16:47:05.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 861cd02e-ed7a-400d-9db9-66582946fb54' 2026-02-25T16:47:05.070 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 861cd02e-ed7a-400d-9db9-66582946fb54 2026-02-25T16:47:05.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:05.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCJJ59pkuLCBBAAH9OgLsmkOH0X6+PYex+CfA== 2026-02-25T16:47:05.080 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCJJ59pkuLCBBAAH9OgLsmkOH0X6+PYex+CfA=="}' 2026-02-25T16:47:05.081 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 861cd02e-ed7a-400d-9db9-66582946fb54 -i td/test-erasure-eio/2/new.json 2026-02-25T16:47:05.310 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:47:05.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:47:05.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJJ59pkuLCBBAAH9OgLsmkOH0X6+PYex+CfA== --osd-uuid 861cd02e-ed7a-400d-9db9-66582946fb54 2026-02-25T16:47:05.332 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:05.331+0000 7f3697239780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:05.334 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:05.332+0000 7f3697239780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:05.335 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:05.333+0000 7f3697239780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:05.335 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:05.333+0000 7f3697239780 -1 bdev(0x3384c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:05.335 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:05.333+0000 7f3697239780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:47:05.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:47:05.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:05.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:47:05.698 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:47:05.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:06.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:47:06.015 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:47:06.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:06.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:06.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:06.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:06.029 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:06.028+0000 7f74a253d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:06.031 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:06.029+0000 7f74a253d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:06.031 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:06.030+0000 7f74a253d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:06.165 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:06.164+0000 7f74a253d780 -1 Falling back to public interface 2026-02-25T16:47:06.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:47:06.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:06.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:47:06.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:06.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:06.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:06.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:06.241 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:06.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:06.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:47:06.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:06.529 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:06.527+0000 7f74a253d780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:47:07.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:07.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:07.470 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:07.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:07.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:07.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:47:07.694 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3192908453,v1:127.0.0.1:6819/3192908453] [v2:127.0.0.1:6820/3192908453,v1:127.0.0.1:6821/3192908453] exists,up 861cd02e-ed7a-400d-9db9-66582946fb54 2026-02-25T16:47:07.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:07.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:07.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:07.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:07.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:07.700 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:07.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:07.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:07.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:47:07.702 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:07.702 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2fdd6e13-f076-4479-a482-53d1ba95d510 2026-02-25T16:47:07.702 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 2fdd6e13-f076-4479-a482-53d1ba95d510 2026-02-25T16:47:07.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 2fdd6e13-f076-4479-a482-53d1ba95d510' 2026-02-25T16:47:07.703 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:07.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCLJ59pgZsYKhAAjWi4XalHs0CynU5OMQjTAA== 2026-02-25T16:47:07.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCLJ59pgZsYKhAAjWi4XalHs0CynU5OMQjTAA=="}' 2026-02-25T16:47:07.707 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2fdd6e13-f076-4479-a482-53d1ba95d510 -i td/test-erasure-eio/3/new.json 2026-02-25T16:47:07.937 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:47:07.944 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:47:07.945 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLJ59pgZsYKhAAjWi4XalHs0CynU5OMQjTAA== --osd-uuid 2fdd6e13-f076-4479-a482-53d1ba95d510 2026-02-25T16:47:07.959 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:07.958+0000 7ff35d84c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:07.961 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:07.959+0000 7ff35d84c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:07.961 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:07.960+0000 7ff35d84c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:07.962 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:07.960+0000 7ff35d84c780 -1 bdev(0x43b6c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:07.962 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:07.960+0000 7ff35d84c780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:47:08.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:47:08.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:08.317 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:47:08.317 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:47:08.318 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:08.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:47:08.635 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:47:08.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:08.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:08.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:08.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:08.648 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:08.646+0000 7f9270a52780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:08.649 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:08.647+0000 7f9270a52780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:08.650 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:08.648+0000 7f9270a52780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:08.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:47:08.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:08.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:47:08.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:08.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:08.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:08.861 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:08.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:08.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:08.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:47:08.863 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:08.862+0000 7f9270a52780 -1 Falling back to public interface 2026-02-25T16:47:09.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:09.170 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:09.168+0000 7f9270a52780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:47:10.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:10.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:10.091 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:10.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:10.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:10.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:47:10.325 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2276896034,v1:127.0.0.1:6827/2276896034] [v2:127.0.0.1:6828/2276896034,v1:127.0.0.1:6829/2276896034] exists,up 2fdd6e13-f076-4479-a482-53d1ba95d510 2026-02-25T16:47:10.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:10.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:47:10.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:47:10.370 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:47:10.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:47:10.376 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:47:01.176+0000 7fb5b6a34780 0 load: jerasure load: lrc 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:278: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:279: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T16:47:10.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:47:10.378 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T16:47:10.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:47:10.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:47:11.027 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:47:11.034 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:47:12.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:47:12.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:47:12.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:47:12.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:47:12.036 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:47:12.037 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:47:12.037 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:47:12.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:47:12.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:47:12.037 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:47:12.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:47:12.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:47:12.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:47:12.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:47:12.127 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:47:12.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:47:12.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:47:12.348 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:12.348 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:47:12.348 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:47:12.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:47:12.348 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:12.348 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:47:12.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803780 2026-02-25T16:47:12.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803780 2026-02-25T16:47:12.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780' 2026-02-25T16:47:12.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:12.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:47:12.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607556 2026-02-25T16:47:12.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607556 2026-02-25T16:47:12.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556' 2026-02-25T16:47:12.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:12.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:47:12.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411331 2026-02-25T16:47:12.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411331 2026-02-25T16:47:12.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556 2-77309411331' 2026-02-25T16:47:12.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:12.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:47:12.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149699 2026-02-25T16:47:12.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149699 2026-02-25T16:47:12.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556 2-77309411331 3-111669149699' 2026-02-25T16:47:12.603 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:12.604 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803780 2026-02-25T16:47:12.604 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:12.604 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:47:12.605 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803780 2026-02-25T16:47:12.605 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:12.606 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803780 2026-02-25T16:47:12.606 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803780' 2026-02-25T16:47:12.606 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803780 2026-02-25T16:47:12.607 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:12.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803778 -lt 25769803780 2026-02-25T16:47:12.832 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:47:13.831 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:47:13.831 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:14.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803780 2026-02-25T16:47:14.058 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:14.058 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607556 2026-02-25T16:47:14.059 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:14.059 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:47:14.060 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607556 2026-02-25T16:47:14.060 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:14.061 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607556 2026-02-25T16:47:14.062 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607556' 2026-02-25T16:47:14.062 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607556 2026-02-25T16:47:14.062 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:47:14.287 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607556 -lt 51539607556 2026-02-25T16:47:14.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:14.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-77309411331 2026-02-25T16:47:14.288 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:14.289 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:47:14.289 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-77309411331 2026-02-25T16:47:14.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:14.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411331 2026-02-25T16:47:14.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 77309411331' 2026-02-25T16:47:14.291 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 77309411331 2026-02-25T16:47:14.291 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:47:14.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411332 -lt 77309411331 2026-02-25T16:47:14.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:14.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-111669149699 2026-02-25T16:47:14.515 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:14.516 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:47:14.516 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-111669149699 2026-02-25T16:47:14.516 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:14.517 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=111669149699 2026-02-25T16:47:14.518 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 111669149699 2026-02-25T16:47:14.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2026-02-25T16:47:14.518 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:47:14.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 111669149699 -lt 111669149699 2026-02-25T16:47:14.745 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:47:14.745 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:14.745 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:15.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:47:15.063 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:47:15.063 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:47:15.064 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:47:15.064 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:47:15.064 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:47:15.064 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:47:15.064 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:47:15.283 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:47:15.284 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:47:15.284 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:15.284 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:47:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:47:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:47:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:281: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2026-02-25T16:47:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-96095-0 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-96095-0 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:47:15.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-96095-0 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:47:15.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-96095-0 td/test-erasure-eio/ORIGINAL 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-96095-0 td/test-erasure-eio 0 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-96095-0 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:47:15.615 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-96095-0 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-0 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-0 2026-02-25T16:47:15.616 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:47:15.846 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:47:15.846 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:47:15.847 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:47:15.848 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:47:15.893 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:47:15.894 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:47:15.895 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:47:15.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T16:47:15.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:47:15.895 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:47:15.895 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:15.895 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:15.896 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:15.896 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:47:15.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:47:15.896 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-96095-0 0 2026-02-25T16:47:15.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-96095-0 2026-02-25T16:47:15.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:47:15.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:47:15.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-96095-0 2026-02-25T16:47:15.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:47:15.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:47:15.942 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-96095-0 td/test-erasure-eio/COPY 2026-02-25T16:47:15.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:47:15.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:47:15.959 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-25T16:47:15.960 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=1 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-96095-0 td/test-erasure-eio 1 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:47:15.962 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-96095-0 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-96095-0 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-0 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-0 2026-02-25T16:47:15.963 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:47:16.188 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T16:47:16.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:47:16.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T16:47:16.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:47:16.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T16:47:16.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:47:16.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:47:16.190 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:47:16.190 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T16:47:16.190 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:47:16.190 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:47:16.190 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:16.190 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:16.190 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:16.191 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:47:16.191 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:47:16.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:47:16.239 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:47:16.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:47:16.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T16:47:16.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:47:16.241 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:47:16.241 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:16.241 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:16.241 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:16.241 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:47:16.241 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:47:16.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-96095-0 1 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 0 -eq 1 -a 1 -eq 2 ']' 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-96095-0 fail 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-96095-0 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-25T16:47:16.288 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-96095-0 td/test-erasure-eio/COPY 2026-02-25T16:47:16.304 INFO:tasks.workunit.client.0.trial129.stderr:error getting pool-jerasure/obj-eio-96095-0: (5) Input/output error 2026-02-25T16:47:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-25T16:47:16.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:47:16.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:47:16.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:47:16.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:47:16.577 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:47:16.585 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:47:16.873 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:47:16.882 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:47:16.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:47:16.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:47:16.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:47:16.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:47:16.883 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:47:16.883 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:47:16.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:47:16.884 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:47:16.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:47:16.991 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:47:16.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:47:16.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:47:16.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:47:16.993 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:47:16.993 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:47:16.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:16.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:47:16.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:47:16.995 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:16.995 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:47:16.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:47:16.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:47:17.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:47:17.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:17.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:17.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:47:17.007 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:47:17.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:47:17.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:47:17.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:47:17.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:47:17.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:47:17.009 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:47:17.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:47:17.010 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:47:17.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:47:17.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:47:17.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:47:17.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:17.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:47:17.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:47:17.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:17.014 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:47:17.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:47:17.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:47:17.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:47:17.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:17.016 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:17.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:47:17.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:47:17.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:47:17.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:47:17.018 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:47:17.018 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:17.018 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:17.018 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:47:17.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:47:17.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:47:17.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:47:17.040 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:47:17.041 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:17.041 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:17.041 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:17.041 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:17.041 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:17.041 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:17.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:47:17.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:47:17.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:47:17.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:47:17.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:47:17.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:47:17.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:47:17.067 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:47:17.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:47:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:47:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:47:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:47:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:47:17.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:47:17.116 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:47:17.116 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:47:17.116 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:47:17.116 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:17.116 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:17.116 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:17.117 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:47:17.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:47:17.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:47:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:47:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:47:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:47:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:47:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:47:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:47:17.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:47:17.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:47:17.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:17.259 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:17.259 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:17.259 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:17.260 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:17.260 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:17.260 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:47:17.261 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:47:17.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:47:17.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:47:17.370 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:47:17.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:47:18.378 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:47:18.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:47:18.417 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:47:18.421 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:47:18.423 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:47:17.056+0000 7fc922c48c00 0 load: jerasure load: lrc 2026-02-25T16:47:18.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2026-02-25T16:47:18.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:287: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2026-02-25T16:47:18.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2026-02-25T16:47:18.424 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T16:47:18.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:47:18.425 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T16:47:18.425 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T16:47:18.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:47:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:47:18.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:47:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:18.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:18.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:18.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=721ee416-e61d-4023-ba81-1565e19efd1f 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 721ee416-e61d-4023-ba81-1565e19efd1f' 2026-02-25T16:47:18.437 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 721ee416-e61d-4023-ba81-1565e19efd1f 2026-02-25T16:47:18.438 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:18.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCWJ59pdS4QGhAAU/GQM83rKNlSUlGBHy2e9w== 2026-02-25T16:47:18.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCWJ59pdS4QGhAAU/GQM83rKNlSUlGBHy2e9w=="}' 2026-02-25T16:47:18.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 721ee416-e61d-4023-ba81-1565e19efd1f -i td/test-erasure-eio/0/new.json 2026-02-25T16:47:18.539 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:18.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:47:18.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWJ59pdS4QGhAAU/GQM83rKNlSUlGBHy2e9w== --osd-uuid 721ee416-e61d-4023-ba81-1565e19efd1f 2026-02-25T16:47:18.561 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:18.560+0000 7ff2ae5fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:18.563 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:18.561+0000 7ff2ae5fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:18.563 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:18.562+0000 7ff2ae5fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:18.564 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:18.562+0000 7ff2ae5fd780 -1 bdev(0x23fcc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:18.564 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:18.562+0000 7ff2ae5fd780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:47:19.011 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:47:19.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:19.012 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:47:19.012 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:47:19.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:19.114 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:47:19.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:47:19.115 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:19.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:19.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:19.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:19.129 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:19.127+0000 7feb0f216780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:19.131 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:19.129+0000 7feb0f216780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:19.131 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:19.130+0000 7feb0f216780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:19.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:47:19.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:19.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:47:19.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:19.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:19.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:19.213 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:19.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:19.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:19.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:47:19.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:19.348 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:19.347+0000 7feb0f216780 -1 Falling back to public interface 2026-02-25T16:47:19.619 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:19.617+0000 7feb0f216780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:47:20.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:20.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:20.310 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:20.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:20.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:20.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:47:20.541 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4032881230,v1:127.0.0.1:6803/4032881230] [v2:127.0.0.1:6804/4032881230,v1:127.0.0.1:6805/4032881230] exists,up 721ee416-e61d-4023-ba81-1565e19efd1f 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:20.545 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:20.546 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:20.547 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=788a0560-8bc8-4968-925e-2129c730468d 2026-02-25T16:47:20.548 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 788a0560-8bc8-4968-925e-2129c730468d' 2026-02-25T16:47:20.549 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 788a0560-8bc8-4968-925e-2129c730468d 2026-02-25T16:47:20.549 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:20.553 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYJ59pqEftIBAALAg1Y00AMj8OvkaMD8vYoQ== 2026-02-25T16:47:20.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYJ59pqEftIBAALAg1Y00AMj8OvkaMD8vYoQ=="}' 2026-02-25T16:47:20.554 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 788a0560-8bc8-4968-925e-2129c730468d -i td/test-erasure-eio/1/new.json 2026-02-25T16:47:20.784 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:20.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:47:20.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYJ59pqEftIBAALAg1Y00AMj8OvkaMD8vYoQ== --osd-uuid 788a0560-8bc8-4968-925e-2129c730468d 2026-02-25T16:47:20.807 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:20.805+0000 7f20cbc29780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:20.809 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:20.807+0000 7f20cbc29780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:20.809 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:20.808+0000 7f20cbc29780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:20.810 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:20.808+0000 7f20cbc29780 -1 bdev(0x244ac00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:20.810 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:20.808+0000 7f20cbc29780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:47:21.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:47:21.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:21.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:47:21.155 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:47:21.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:21.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:47:21.471 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:47:21.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:21.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:21.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:21.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:21.485 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:21.483+0000 7f3108e33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:21.486 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:21.484+0000 7f3108e33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:21.487 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:21.485+0000 7f3108e33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:21.634 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:21.633+0000 7f3108e33780 -1 Falling back to public interface 2026-02-25T16:47:21.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:47:21.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:21.703 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:47:21.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:21.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:21.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:21.704 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:21.704 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:21.705 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:21.705 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:21.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:21.957 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:21.955+0000 7f3108e33780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:47:22.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:22.930 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:22.930 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:22.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:22.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:22.931 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:23.156 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1407421462,v1:127.0.0.1:6811/1407421462] [v2:127.0.0.1:6812/1407421462,v1:127.0.0.1:6813/1407421462] exists,up 788a0560-8bc8-4968-925e-2129c730468d 2026-02-25T16:47:23.160 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:23.161 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:23.162 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:23.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:23.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:23.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:47:23.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:23.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bb5ebc42-db7d-4a7e-abc8-3fec7445b0e4 2026-02-25T16:47:23.164 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 bb5ebc42-db7d-4a7e-abc8-3fec7445b0e4' 2026-02-25T16:47:23.164 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 bb5ebc42-db7d-4a7e-abc8-3fec7445b0e4 2026-02-25T16:47:23.164 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:23.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCbJ59pPIIPChAA1clNOARYsEN/sBGgvin6hw== 2026-02-25T16:47:23.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCbJ59pPIIPChAA1clNOARYsEN/sBGgvin6hw=="}' 2026-02-25T16:47:23.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bb5ebc42-db7d-4a7e-abc8-3fec7445b0e4 -i td/test-erasure-eio/2/new.json 2026-02-25T16:47:23.405 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:47:23.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:47:23.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbJ59pPIIPChAA1clNOARYsEN/sBGgvin6hw== --osd-uuid bb5ebc42-db7d-4a7e-abc8-3fec7445b0e4 2026-02-25T16:47:23.430 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:23.428+0000 7fef24821780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:23.431 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:23.430+0000 7fef24821780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:23.432 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:23.430+0000 7fef24821780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:23.433 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:23.431+0000 7fef24821780 -1 bdev(0x4118c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:23.433 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:23.431+0000 7fef24821780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:47:23.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:47:23.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:23.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:47:23.793 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:47:23.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:24.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:47:24.108 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:47:24.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:24.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:24.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:24.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:24.122 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:24.120+0000 7fc0b6208780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:24.123 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:24.121+0000 7fc0b6208780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:24.124 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:24.122+0000 7fc0b6208780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:24.304 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:24.303+0000 7fc0b6208780 -1 Falling back to public interface 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:24.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:47:24.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:24.660 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:24.658+0000 7fc0b6208780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:47:25.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:25.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:25.560 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:25.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:25.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:25.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:47:25.786 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2334159561,v1:127.0.0.1:6819/2334159561] [v2:127.0.0.1:6820/2334159561,v1:127.0.0.1:6821/2334159561] exists,up bb5ebc42-db7d-4a7e-abc8-3fec7445b0e4 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:47:25.787 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:25.788 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:47:25.789 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:25.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=22407d35-86ee-4d5d-814d-850bef143163 2026-02-25T16:47:25.790 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 22407d35-86ee-4d5d-814d-850bef143163' 2026-02-25T16:47:25.790 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 22407d35-86ee-4d5d-814d-850bef143163 2026-02-25T16:47:25.790 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:25.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdJ59pl9abLxAAe9vd543AzORG+8/q7qVAjQ== 2026-02-25T16:47:25.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdJ59pl9abLxAAe9vd543AzORG+8/q7qVAjQ=="}' 2026-02-25T16:47:25.799 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 22407d35-86ee-4d5d-814d-850bef143163 -i td/test-erasure-eio/3/new.json 2026-02-25T16:47:26.028 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:47:26.035 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:47:26.036 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdJ59pl9abLxAAe9vd543AzORG+8/q7qVAjQ== --osd-uuid 22407d35-86ee-4d5d-814d-850bef143163 2026-02-25T16:47:26.051 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.049+0000 7f47b780e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:26.052 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.051+0000 7f47b780e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:26.053 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.052+0000 7f47b780e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:26.053 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.052+0000 7f47b780e780 -1 bdev(0x4252c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:26.054 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.052+0000 7f47b780e780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:47:26.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:47:26.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:26.403 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:47:26.403 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:47:26.404 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:26.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:47:26.722 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:47:26.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:26.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:26.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:26.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:26.735 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.734+0000 7fa88d464780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:26.737 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.735+0000 7fa88d464780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:26.737 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.736+0000 7fa88d464780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:26.866 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:26.864+0000 7fa88d464780 -1 Falling back to public interface 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:26.946 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:47:27.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:27.215 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:27.213+0000 7fa88d464780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:47:28.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:28.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:28.176 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:28.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:28.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:28.176 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:47:28.405 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3176667203,v1:127.0.0.1:6827/3176667203] [v2:127.0.0.1:6828/3176667203,v1:127.0.0.1:6829/3176667203] exists,up 22407d35-86ee-4d5d-814d-850bef143163 2026-02-25T16:47:28.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:28.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:28.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:47:28.406 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:47:28.451 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:47:28.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:47:28.457 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:47:19.349+0000 7feb0f216780 0 load: jerasure load: lrc 2026-02-25T16:47:28.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2026-02-25T16:47:28.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:291: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T16:47:28.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:47:28.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:47:28.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T16:47:28.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:47:28.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T16:47:28.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:47:28.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T16:47:28.744 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:47:28.745 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:47:29.116 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:47:29.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:47:30.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:47:30.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:47:30.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:47:30.125 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:47:30.126 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:47:30.127 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:47:30.127 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:47:30.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:47:30.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:47:30.127 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:47:30.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:47:30.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:47:30.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:47:30.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:47:30.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:47:30.214 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:47:30.435 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:47:30.436 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:30.436 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:47:30.436 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:47:30.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:47:30.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:30.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:47:30.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803780 2026-02-25T16:47:30.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803780 2026-02-25T16:47:30.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780' 2026-02-25T16:47:30.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:30.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:47:30.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574853 2026-02-25T16:47:30.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574853 2026-02-25T16:47:30.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574853' 2026-02-25T16:47:30.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:30.565 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:47:30.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378628 2026-02-25T16:47:30.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378628 2026-02-25T16:47:30.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574853 2-81604378628' 2026-02-25T16:47:30.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:30.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:47:30.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116995 2026-02-25T16:47:30.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116995 2026-02-25T16:47:30.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574853 2-81604378628 3-115964116995' 2026-02-25T16:47:30.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:30.693 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803780 2026-02-25T16:47:30.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:30.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:47:30.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803780 2026-02-25T16:47:30.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:30.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803780 2026-02-25T16:47:30.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803780' 2026-02-25T16:47:30.696 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803780 2026-02-25T16:47:30.697 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:30.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803778 -lt 25769803780 2026-02-25T16:47:30.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:47:31.922 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:47:31.923 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:32.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803780 2026-02-25T16:47:32.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:32.151 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574853 2026-02-25T16:47:32.151 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:32.151 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:47:32.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574853 2026-02-25T16:47:32.152 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:32.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574853 2026-02-25T16:47:32.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574853' 2026-02-25T16:47:32.154 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574853 2026-02-25T16:47:32.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:47:32.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574853 -lt 55834574853 2026-02-25T16:47:32.381 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:32.381 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378628 2026-02-25T16:47:32.381 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:32.382 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:47:32.382 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378628 2026-02-25T16:47:32.383 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:32.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378628 2026-02-25T16:47:32.384 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378628' 2026-02-25T16:47:32.384 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378628 2026-02-25T16:47:32.384 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:47:32.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378629 -lt 81604378628 2026-02-25T16:47:32.609 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:32.610 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116995 2026-02-25T16:47:32.610 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:32.610 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:47:32.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116995 2026-02-25T16:47:32.611 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:32.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116995 2026-02-25T16:47:32.612 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116995' 2026-02-25T16:47:32.612 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116995 2026-02-25T16:47:32.613 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:47:32.836 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116995 -lt 115964116995 2026-02-25T16:47:32.838 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:47:32.838 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:32.838 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:33.153 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:47:33.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:47:33.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:47:33.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:47:33.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:47:33.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:47:33.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:47:33.154 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:47:33.382 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:47:33.382 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:47:33.382 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:33.382 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:33.698 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:293: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:294: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T16:47:33.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-96095-1 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-96095-1 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-96095-1 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:47:33.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-96095-1 td/test-erasure-eio/ORIGINAL 2026-02-25T16:47:33.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-96095-1 td/test-erasure-eio 1 2026-02-25T16:47:33.716 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-96095-1 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:47:33.717 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T16:47:33.718 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:47:33.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-96095-1 2026-02-25T16:47:33.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:47:33.718 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-1 2026-02-25T16:47:33.718 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-1 2026-02-25T16:47:33.718 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:47:33.950 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:47:33.950 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:33.950 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:47:33.950 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:47:33.950 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T16:47:33.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:47:33.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T16:47:33.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:47:33.951 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T16:47:33.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:47:33.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T16:47:33.951 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:47:33.952 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:33.953 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:33.953 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:33.953 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:47:33.953 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:47:33.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:47:33.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:47:33.997 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:47:33.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T16:47:33.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:47:33.997 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:47:33.998 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:33.998 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:33.998 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:33.998 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:47:33.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:47:33.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-96095-1 1 2026-02-25T16:47:34.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-96095-1 2026-02-25T16:47:34.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:47:34.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:47:34.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-96095-1 2026-02-25T16:47:34.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:47:34.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:47:34.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-96095-1 td/test-erasure-eio/COPY 2026-02-25T16:47:34.064 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:47:34.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:47:34.065 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-25T16:47:34.066 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-96095-1 td/test-erasure-eio 2 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-96095-1 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=2 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-96095-1 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:47:34.067 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-1 2026-02-25T16:47:34.068 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-1 2026-02-25T16:47:34.068 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:47:34.301 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:47:34.301 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:34.301 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:47:34.301 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:47:34.301 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T16:47:34.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:47:34.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=2 2026-02-25T16:47:34.302 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:47:34.302 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/2/type 2026-02-25T16:47:34.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:47:34.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2026-02-25T16:47:34.304 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=2 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.2 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:34.305 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:34.306 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.2.asok 2026-02-25T16:47:34.306 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:47:34.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:47:34.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.2 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.2.asok 2026-02-25T16:47:34.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:47:34.353 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-96095-1 2 2026-02-25T16:47:34.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T16:47:34.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-96095-1 2026-02-25T16:47:34.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:47:34.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:47:34.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-96095-1 2026-02-25T16:47:34.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:47:34.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:47:34.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-96095-1 td/test-erasure-eio/COPY 2026-02-25T16:47:34.414 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:47:34.415 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:47:34.416 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:47:34.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:295: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:47:34.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:47:34.417 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:47:34.687 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:47:34.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:47:34.969 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:47:34.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:47:34.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:47:34.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:47:34.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:47:34.978 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:47:34.978 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:47:34.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:47:34.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:47:34.978 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:47:35.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:47:35.086 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:47:35.086 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:47:35.087 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:47:35.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:47:35.088 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:47:35.089 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:47:35.089 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:35.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:47:35.090 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:47:35.090 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:35.091 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:47:35.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:47:35.092 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:47:35.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:47:35.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:35.101 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:35.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:47:35.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:47:35.104 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:47:35.104 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:47:35.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:47:35.105 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:47:35.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:47:35.106 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:47:35.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:47:35.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:47:35.107 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:47:35.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:35.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:47:35.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:47:35.109 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:47:35.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:47:35.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:47:35.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:47:35.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:47:35.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:35.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:35.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:47:35.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:47:35.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:47:35.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:47:35.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:47:35.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:35.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:35.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:47:35.115 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:47:35.116 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:47:35.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:47:35.134 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:47:35.134 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:35.134 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:35.134 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:35.134 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:35.134 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:35.135 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:35.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:47:35.157 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:35.158 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:35.159 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:47:35.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:47:35.159 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:47:35.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:47:35.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:47:35.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:35.203 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:47:35.204 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:47:35.204 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:47:35.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:47:35.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:47:35.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:47:35.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:47:35.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:47:35.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:47:35.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:47:35.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:47:35.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:35.342 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:35.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:35.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:35.343 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:35.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:35.343 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:47:35.344 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:47:35.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:47:35.358 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:47:35.453 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:47:35.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:47:36.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:47:36.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:47:36.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:47:36.461 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:36.461 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:36.461 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:36.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:47:36.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:47:36.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:47:36.501 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:47:36.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:47:36.506 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:47:35.148+0000 7f68c065ac00 0 load: jerasure load: lrc 2026-02-25T16:47:36.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2026-02-25T16:47:36.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2026-02-25T16:47:36.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_subread_missing: setup_osds 4 2026-02-25T16:47:36.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T16:47:36.507 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:47:36.507 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T16:47:36.508 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T16:47:36.511 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:47:36.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:36.513 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:47:36.514 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:36.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=423ce78d-f0fb-4bf9-a519-5659c495a5fa 2026-02-25T16:47:36.515 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 423ce78d-f0fb-4bf9-a519-5659c495a5fa' 2026-02-25T16:47:36.515 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 423ce78d-f0fb-4bf9-a519-5659c495a5fa 2026-02-25T16:47:36.516 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:36.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoJ59pvjANHxAAm67P1BgfCgOIucj34/El9Q== 2026-02-25T16:47:36.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoJ59pvjANHxAAm67P1BgfCgOIucj34/El9Q=="}' 2026-02-25T16:47:36.522 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 423ce78d-f0fb-4bf9-a519-5659c495a5fa -i td/test-erasure-eio/0/new.json 2026-02-25T16:47:36.624 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:36.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:47:36.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoJ59pvjANHxAAm67P1BgfCgOIucj34/El9Q== --osd-uuid 423ce78d-f0fb-4bf9-a519-5659c495a5fa 2026-02-25T16:47:36.647 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:36.646+0000 7f39fb45a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:36.649 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:36.647+0000 7f39fb45a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:36.650 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:36.648+0000 7f39fb45a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:36.650 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:36.648+0000 7f39fb45a780 -1 bdev(0x2b60c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:36.650 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:36.648+0000 7f39fb45a780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:47:37.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:47:37.109 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:37.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:47:37.110 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:47:37.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:37.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:47:37.212 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:47:37.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:37.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:37.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:37.213 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:37.227 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:37.225+0000 7f1026435780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:37.228 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:37.226+0000 7f1026435780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:37.229 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:37.227+0000 7f1026435780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:37.308 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:47:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:47:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:37.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:37.309 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:37.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:37.310 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:47:37.405 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:37.437 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:37.435+0000 7f1026435780 -1 Falling back to public interface 2026-02-25T16:47:37.710 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:37.708+0000 7f1026435780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:47:38.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:38.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:38.407 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:38.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:38.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:38.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:47:38.634 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1768370947,v1:127.0.0.1:6803/1768370947] [v2:127.0.0.1:6804/1768370947,v1:127.0.0.1:6805/1768370947] exists,up 423ce78d-f0fb-4bf9-a519-5659c495a5fa 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:38.635 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:38.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:38.637 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:47:38.637 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:38.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=473567ac-b31e-4b00-80ec-5d0fdcdb0d96 2026-02-25T16:47:38.638 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 473567ac-b31e-4b00-80ec-5d0fdcdb0d96' 2026-02-25T16:47:38.638 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 473567ac-b31e-4b00-80ec-5d0fdcdb0d96 2026-02-25T16:47:38.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:38.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCqJ59pBlieJhAAX0E+LAE7T3VEGPYVxRmXsw== 2026-02-25T16:47:38.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCqJ59pBlieJhAAX0E+LAE7T3VEGPYVxRmXsw=="}' 2026-02-25T16:47:38.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 473567ac-b31e-4b00-80ec-5d0fdcdb0d96 -i td/test-erasure-eio/1/new.json 2026-02-25T16:47:38.880 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:38.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:47:38.888 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqJ59pBlieJhAAX0E+LAE7T3VEGPYVxRmXsw== --osd-uuid 473567ac-b31e-4b00-80ec-5d0fdcdb0d96 2026-02-25T16:47:38.904 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:38.902+0000 7fe3ae80b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:38.905 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:38.903+0000 7fe3ae80b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:38.906 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:38.904+0000 7fe3ae80b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:38.906 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:38.904+0000 7fe3ae80b780 -1 bdev(0x36bac00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:38.906 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:38.904+0000 7fe3ae80b780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:47:39.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:47:39.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:39.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:47:39.244 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:47:39.244 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:39.568 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:47:39.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:47:39.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:39.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:39.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:39.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:39.583 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:39.582+0000 7f37fa632780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:39.584 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:39.583+0000 7f37fa632780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:39.585 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:39.584+0000 7f37fa632780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:39.747 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:39.746+0000 7f37fa632780 -1 Falling back to public interface 2026-02-25T16:47:39.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:47:39.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:39.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:47:39.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:39.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:39.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:39.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:39.797 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:39.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:39.798 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:40.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:40.121 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:40.119+0000 7f37fa632780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:47:41.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:41.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:41.023 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:41.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:41.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:41.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:41.248 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/880444936,v1:127.0.0.1:6811/880444936] [v2:127.0.0.1:6812/880444936,v1:127.0.0.1:6813/880444936] exists,up 473567ac-b31e-4b00-80ec-5d0fdcdb0d96 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:41.249 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:41.250 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:47:41.251 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:41.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6e805a93-e408-4fe5-84f3-6c4e6eaa0ace 2026-02-25T16:47:41.252 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6e805a93-e408-4fe5-84f3-6c4e6eaa0ace' 2026-02-25T16:47:41.252 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 6e805a93-e408-4fe5-84f3-6c4e6eaa0ace 2026-02-25T16:47:41.253 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:41.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCtJ59pa4OZDxAANI4Q8bGorfI9Arpb31ATmg== 2026-02-25T16:47:41.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCtJ59pa4OZDxAANI4Q8bGorfI9Arpb31ATmg=="}' 2026-02-25T16:47:41.262 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6e805a93-e408-4fe5-84f3-6c4e6eaa0ace -i td/test-erasure-eio/2/new.json 2026-02-25T16:47:41.488 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:47:41.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:47:41.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtJ59pa4OZDxAANI4Q8bGorfI9Arpb31ATmg== --osd-uuid 6e805a93-e408-4fe5-84f3-6c4e6eaa0ace 2026-02-25T16:47:41.513 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:41.511+0000 7f76d1e31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:41.514 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:41.513+0000 7f76d1e31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:41.515 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:41.514+0000 7f76d1e31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:41.516 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:41.514+0000 7f76d1e31780 -1 bdev(0x3e52c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:41.516 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:41.514+0000 7f76d1e31780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:47:41.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:47:41.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:41.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:47:41.855 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:47:41.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:42.170 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:47:42.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:47:42.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:42.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:42.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:42.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:42.183 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:42.182+0000 7f9588266780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:42.185 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:42.183+0000 7f9588266780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:42.185 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:42.184+0000 7f9588266780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:42.330 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:42.329+0000 7f9588266780 -1 Falling back to public interface 2026-02-25T16:47:42.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:47:42.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:42.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:47:42.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:42.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:42.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:42.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:42.400 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:42.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:42.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:47:42.622 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:42.642 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:42.641+0000 7f9588266780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:47:43.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:43.624 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:43.625 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:43.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:43.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:43.626 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:47:43.857 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1492141545,v1:127.0.0.1:6819/1492141545] [v2:127.0.0.1:6820/1492141545,v1:127.0.0.1:6821/1492141545] exists,up 6e805a93-e408-4fe5-84f3-6c4e6eaa0ace 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:47:43.858 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:47:43.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:47:43.861 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:47:43.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1a353212-2a6c-40dd-ac9d-96cd57677589 2026-02-25T16:47:43.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1a353212-2a6c-40dd-ac9d-96cd57677589' 2026-02-25T16:47:43.862 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 1a353212-2a6c-40dd-ac9d-96cd57677589 2026-02-25T16:47:43.862 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:47:43.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCvJ59po0jaMxAAFVXD6pTxiRW2rXZlkhOEYA== 2026-02-25T16:47:43.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCvJ59po0jaMxAAFVXD6pTxiRW2rXZlkhOEYA=="}' 2026-02-25T16:47:43.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1a353212-2a6c-40dd-ac9d-96cd57677589 -i td/test-erasure-eio/3/new.json 2026-02-25T16:47:44.098 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:47:44.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:47:44.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvJ59po0jaMxAAFVXD6pTxiRW2rXZlkhOEYA== --osd-uuid 1a353212-2a6c-40dd-ac9d-96cd57677589 2026-02-25T16:47:44.121 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.119+0000 7fab99235780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:44.122 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.121+0000 7fab99235780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:44.124 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.122+0000 7fab99235780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:44.124 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.122+0000 7fab99235780 -1 bdev(0x2a3ec00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:47:44.124 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.122+0000 7fab99235780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:47:44.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:47:44.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:47:44.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:47:44.471 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:47:44.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:47:44.792 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:47:44.792 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:47:44.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:44.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:47:44.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:47:44.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:47:44.807 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.805+0000 7f5b17211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:44.808 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.806+0000 7f5b17211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:44.809 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.807+0000 7f5b17211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:44.949 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:44.948+0000 7f5b17211780 -1 Falling back to public interface 2026-02-25T16:47:45.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:47:45.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:45.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:47:45.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:45.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:45.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:45.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:45.017 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:47:45.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:45.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:47:45.243 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:45.369 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:45.367+0000 7f5b17211780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:47:46.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:46.245 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:46.245 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:47:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:46.246 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:47:46.473 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2488823751,v1:127.0.0.1:6827/2488823751] [v2:127.0.0.1:6828/2488823751,v1:127.0.0.1:6829/2488823751] exists,up 1a353212-2a6c-40dd-ac9d-96cd57677589 2026-02-25T16:47:46.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:46.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:46.475 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:46.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:47:46.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:47:46.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:47:46.475 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:47:46.476 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:46.476 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:46.476 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:47:46.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:47:46.476 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:47:46.520 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:47:46.524 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:47:46.526 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:47:37.438+0000 7f1026435780 0 load: jerasure load: lrc 2026-02-25T16:47:46.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2026-02-25T16:47:46.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T16:47:46.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:47:46.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:47:46.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T16:47:46.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:47:46.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T16:47:46.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:47:46.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T16:47:46.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:47:46.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:47:47.190 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:47:47.197 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:47:48.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:47:48.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:47:48.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:47:48.199 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:47:48.200 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:47:48.200 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:47:48.201 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:47:48.201 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:47:48.201 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:47:48.201 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:47:48.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:47:48.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:47:48.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:47:48.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:47:48.291 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:47:48.291 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:47:48.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:47:48.520 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:48.520 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:47:48.520 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:47:48.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:47:48.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:48.521 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:47:48.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803780 2026-02-25T16:47:48.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803780 2026-02-25T16:47:48.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780' 2026-02-25T16:47:48.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:48.585 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:47:48.649 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-25T16:47:48.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-25T16:47:48.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574852' 2026-02-25T16:47:48.650 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:48.650 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:47:48.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378628 2026-02-25T16:47:48.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378628 2026-02-25T16:47:48.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574852 2-81604378628' 2026-02-25T16:47:48.714 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:48.714 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:47:48.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116996 2026-02-25T16:47:48.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116996 2026-02-25T16:47:48.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-55834574852 2-81604378628 3-115964116996' 2026-02-25T16:47:48.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:48.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803780 2026-02-25T16:47:48.781 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:48.782 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:47:48.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803780 2026-02-25T16:47:48.783 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:48.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803780 2026-02-25T16:47:48.784 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803780' 2026-02-25T16:47:48.784 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803780 2026-02-25T16:47:48.784 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:49.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803778 -lt 25769803780 2026-02-25T16:47:49.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:47:50.010 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:47:50.011 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:50.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803780 2026-02-25T16:47:50.237 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:50.237 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574852 2026-02-25T16:47:50.237 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:50.238 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:47:50.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574852 2026-02-25T16:47:50.239 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:50.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574852 2026-02-25T16:47:50.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574852' 2026-02-25T16:47:50.240 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574852 2026-02-25T16:47:50.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:47:50.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-25T16:47:50.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:50.471 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378628 2026-02-25T16:47:50.472 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:50.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:47:50.473 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378628 2026-02-25T16:47:50.473 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:50.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378628 2026-02-25T16:47:50.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378628' 2026-02-25T16:47:50.474 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378628 2026-02-25T16:47:50.475 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:47:50.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378629 -lt 81604378628 2026-02-25T16:47:50.697 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:50.698 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116996 2026-02-25T16:47:50.698 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:50.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:47:50.699 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116996 2026-02-25T16:47:50.699 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:50.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116996 2026-02-25T16:47:50.701 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116996' 2026-02-25T16:47:50.701 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116996 2026-02-25T16:47:50.701 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:47:50.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116996 -lt 115964116996 2026-02-25T16:47:50.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:47:50.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:50.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:47:51.240 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:47:51.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:47:51.462 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:47:51.462 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:51.462 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:51.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:47:51.778 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_subread_missing: local shard_id=1 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-25T16:47:51.779 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-96095-1 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-96095-1 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-96095-1 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:51.780 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:47:51.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:51.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:47:51.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:47:51.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:47:51.781 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-96095-1 td/test-erasure-eio/ORIGINAL 2026-02-25T16:47:51.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-96095-1 td/test-erasure-eio 1 2026-02-25T16:47:51.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2026-02-25T16:47:51.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2026-02-25T16:47:51.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-96095-1 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-96095-1 2026-02-25T16:47:51.796 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:47:51.797 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-96095-1 2026-02-25T16:47:51.797 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-96095-1 2026-02-25T16:47:51.797 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2026-02-25T16:47:52.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-96095-1 remove 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-96095-1 remove 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:47:52.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-25T16:47:52.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:47:52.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:47:52.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:47:52.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:47:52.027 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-96095-1 remove 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-25T16:47:52.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-96095-1 remove 2026-02-25T16:47:52.443 INFO:tasks.workunit.client.0.trial129.stdout:remove 1#2:c22b4d9f:::obj-remove-96095-1:head# 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:52.490 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:47:52.491 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:47:52.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-25T16:47:52.492 INFO:tasks.workunit.client.0.trial129.stderr:start osd.1 2026-02-25T16:47:52.492 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:52.492 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-25T16:47:52.493 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-25T16:47:52.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:47:52.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:47:52.494 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:47:52.505 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:52.504+0000 7f30e905d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:52.507 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:52.506+0000 7f30e905d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:52.508 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:52.506+0000 7f30e905d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:52.646 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:52.645+0000 7f30e905d780 -1 Falling back to public interface 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:52.719 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:47:52.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:52.720 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:52.956 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:53.051 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:53.049+0000 7f30e905d780 -1 osd.1 35 log_to_monitors true 2026-02-25T16:47:53.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:53.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:53.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:53.957 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:53.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:53.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [13,35) [v2:127.0.0.1:6810/685782473,v1:127.0.0.1:6811/685782473] [v2:127.0.0.1:6812/685782473,v1:127.0.0.1:6813/685782473] exists,up 473567ac-b31e-4b00-80ec-5d0fdcdb0d96 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:47:54.185 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:47:54.186 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:47:54.186 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:47:54.186 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:47:54.186 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:47:54.186 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:47:54.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:47:54.275 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:47:54.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:47:54.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:47:54.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:47:54.276 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:47:54.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:47:54.503 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:54.503 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:47:54.503 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:47:54.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:47:54.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:54.503 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:47:54.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803784 2026-02-25T16:47:54.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803784 2026-02-25T16:47:54.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784' 2026-02-25T16:47:54.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:54.568 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:47:54.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724547 2026-02-25T16:47:54.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724547 2026-02-25T16:47:54.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-167503724547' 2026-02-25T16:47:54.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:54.631 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:47:54.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378632 2026-02-25T16:47:54.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378632 2026-02-25T16:47:54.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-167503724547 2-81604378632' 2026-02-25T16:47:54.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:47:54.693 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:47:54.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-25T16:47:54.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-25T16:47:54.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-167503724547 2-81604378632 3-115964116999' 2026-02-25T16:47:54.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:54.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803784 2026-02-25T16:47:54.757 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:54.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:47:54.759 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803784 2026-02-25T16:47:54.759 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:54.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803784 2026-02-25T16:47:54.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-25T16:47:54.760 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803784 2026-02-25T16:47:54.760 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:54.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803784 2026-02-25T16:47:54.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:47:55.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:47:55.987 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:47:56.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-25T16:47:56.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:56.211 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-167503724547 2026-02-25T16:47:56.211 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:56.212 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:47:56.213 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724547 2026-02-25T16:47:56.213 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:56.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724547 2026-02-25T16:47:56.214 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 167503724547' 2026-02-25T16:47:56.214 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 167503724547 2026-02-25T16:47:56.214 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:47:56.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724547 -lt 167503724547 2026-02-25T16:47:56.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:56.442 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378632 2026-02-25T16:47:56.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:56.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:47:56.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378632 2026-02-25T16:47:56.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:56.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378632 2026-02-25T16:47:56.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378632' 2026-02-25T16:47:56.445 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 81604378632 2026-02-25T16:47:56.445 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:47:56.670 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378632 -lt 81604378632 2026-02-25T16:47:56.671 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:47:56.671 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116999 2026-02-25T16:47:56.671 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:47:56.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:47:56.673 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116999 2026-02-25T16:47:56.673 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:47:56.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116999 2026-02-25T16:47:56.674 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116999' 2026-02-25T16:47:56.674 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964116999 2026-02-25T16:47:56.674 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:47:56.900 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116999 -lt 115964116999 2026-02-25T16:47:56.900 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:47:56.900 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:56.901 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:57.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:47:57.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:47:57.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:47:57.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:47:57.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:47:57.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:47:57.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:47:57.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:47:57.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:47:57.440 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:47:57.440 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:47:57.440 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-96095-1 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-96095-1 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:47:57.755 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-96095-1 td/test-erasure-eio/COPY 2026-02-25T16:47:57.771 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:47:57.772 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:47:57.773 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-25T16:47:57.773 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2026-02-25T16:47:57.774 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: local another_shard_id=2 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-96095-1 td/test-erasure-eio 2 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-96095-1 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-96095-1 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-remove-96095-1 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-remove-96095-1 2026-02-25T16:47:57.775 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:47:58.007 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:47:58.007 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:58.007 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:47:58.007 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:47:58.007 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-96095-1 remove 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/test-erasure-eio 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-96095-1 remove 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:47:58.008 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:47:58.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:47:58.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:47:58.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:47:58.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-96095-1 remove 2026-02-25T16:47:58.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-25T16:47:58.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T16:47:58.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-25T16:47:58.315 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T16:47:58.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-25T16:47:58.316 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-96095-1 remove 2026-02-25T16:47:58.422 INFO:tasks.workunit.client.0.trial129.stdout:remove 2#2:c22b4d9f:::obj-remove-96095-1:head# 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:47:58.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:47:58.470 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:58.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:47:58.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:47:58.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:47:58.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:47:58.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:47:58.471 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:start osd.2 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:47:58.472 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:47:58.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:47:58.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:47:58.473 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:47:58.474 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:47:58.485 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:58.483+0000 7fdee7c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:58.486 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:58.485+0000 7fdee7c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:58.487 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:58.485+0000 7fdee7c32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:47:58.629 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:58.627+0000 7fdee7c32780 -1 Falling back to public interface 2026-02-25T16:47:58.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:47:58.699 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:0 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:58.700 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:47:58.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:47:58.956 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:47:58.955+0000 7fdee7c32780 -1 osd.2 41 log_to_monitors true 2026-02-25T16:47:59.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:47:59.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:47:59.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:47:59.927 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:47:59.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:47:59.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:00.154 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 up in weight 1 up_from 44 up_thru 36 down_at 41 last_clean_interval [19,41) [v2:127.0.0.1:6818/1600145821,v1:127.0.0.1:6819/1600145821] [v2:127.0.0.1:6820/1600145821,v1:127.0.0.1:6821/1600145821] exists,up 6e805a93-e408-4fe5-84f3-6c4e6eaa0ace 2026-02-25T16:48:00.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:00.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:00.154 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:00.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T16:48:00.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:48:00.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:48:00.155 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:48:00.155 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:48:00.155 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:48:00.155 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:48:00.156 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:48:00.156 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:48:00.156 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:48:00.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:48:00.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:48:00.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:48:00.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:48:00.242 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:48:00.242 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:48:00.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:48:00.467 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:00.467 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:48:00.467 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:48:00.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:48:00.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:00.468 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:48:00.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803787 2026-02-25T16:48:00.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803787 2026-02-25T16:48:00.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787' 2026-02-25T16:48:00.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:00.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:48:00.593 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724550 2026-02-25T16:48:00.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724550 2026-02-25T16:48:00.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-167503724550' 2026-02-25T16:48:00.594 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:00.594 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:48:00.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561027 2026-02-25T16:48:00.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561027 2026-02-25T16:48:00.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-167503724550 2-188978561027' 2026-02-25T16:48:00.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:00.659 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:48:00.722 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964117002 2026-02-25T16:48:00.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964117002 2026-02-25T16:48:00.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803787 1-167503724550 2-188978561027 3-115964117002' 2026-02-25T16:48:00.723 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:00.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803787 2026-02-25T16:48:00.723 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:00.724 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:48:00.724 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803787 2026-02-25T16:48:00.724 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:00.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803787 2026-02-25T16:48:00.725 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2026-02-25T16:48:00.726 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.0 seq 25769803787 2026-02-25T16:48:00.726 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:00.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803787 2026-02-25T16:48:00.953 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:48:01.954 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:48:01.954 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:02.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803787 -lt 25769803787 2026-02-25T16:48:02.178 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:02.178 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-167503724550 2026-02-25T16:48:02.178 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:02.179 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:48:02.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724550 2026-02-25T16:48:02.179 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:02.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=167503724550 2026-02-25T16:48:02.181 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2026-02-25T16:48:02.181 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.1 seq 167503724550 2026-02-25T16:48:02.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:48:02.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 167503724550 -lt 167503724550 2026-02-25T16:48:02.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:02.409 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-188978561027 2026-02-25T16:48:02.409 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:02.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:48:02.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-188978561027 2026-02-25T16:48:02.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:02.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561027 2026-02-25T16:48:02.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 188978561027' 2026-02-25T16:48:02.412 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.2 seq 188978561027 2026-02-25T16:48:02.412 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:48:02.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561027 -lt 188978561027 2026-02-25T16:48:02.636 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:02.636 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964117002 2026-02-25T16:48:02.637 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:02.637 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:48:02.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964117002 2026-02-25T16:48:02.638 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:02.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964117002 2026-02-25T16:48:02.639 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964117002' 2026-02-25T16:48:02.639 INFO:tasks.workunit.client.0.trial129.stderr:waiting osd.3 seq 115964117002 2026-02-25T16:48:02.640 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:48:02.864 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117002 -lt 115964117002 2026-02-25T16:48:02.864 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:48:02.864 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:02.864 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:03.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:48:03.180 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:48:03.180 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:48:03.180 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:48:03.180 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:48:03.180 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:48:03.180 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:48:03.181 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:48:03.399 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:48:03.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:48:03.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:03.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:03.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:48:03.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:48:03.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:48:03.712 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: '[' 1 -eq 1 -a 2 -eq 2 ']' 2026-02-25T16:48:03.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:188: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-96095-1 2026-02-25T16:48:03.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:48:03.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:48:03.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-96095-1 2026-02-25T16:48:03.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:48:03.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:48:03.713 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-96095-1 td/test-erasure-eio/COPY 2026-02-25T16:48:03.727 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:48:03.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:48:03.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-25T16:48:03.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:48:03.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:48:03.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:48:04.001 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:48:04.009 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:48:04.290 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:48:04.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:48:04.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:48:04.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:48:04.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:48:04.299 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:04.299 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:04.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:04.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:04.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:04.407 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:04.407 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:48:04.408 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:48:04.408 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:48:04.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:48:04.409 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:48:04.409 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:48:04.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:04.410 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:48:04.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:48:04.411 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:04.411 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:04.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:48:04.412 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:48:04.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:48:04.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:04.423 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:04.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:48:04.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:48:04.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:48:04.425 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:04.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:04.427 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:04.427 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:48:04.427 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:48:04.428 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:48:04.428 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:48:04.429 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:48:04.429 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:48:04.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:04.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:48:04.430 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:48:04.430 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:04.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:04.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:48:04.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:48:04.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:48:04.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:04.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:04.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:48:04.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:48:04.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:48:04.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:48:04.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:48:04.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:04.436 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:04.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:48:04.437 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:48:04.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:48:04.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:48:04.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:48:04.438 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:48:04.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:48:04.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:48:04.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:48:04.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:48:04.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:48:04.439 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:48:04.457 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:48:04.457 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:04.457 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:04.457 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:04.457 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:04.457 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:04.457 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:04.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:48:04.480 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:48:04.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:48:04.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:48:04.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:48:04.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:48:04.482 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:48:04.482 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:48:04.483 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:48:04.483 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:48:04.484 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:04.484 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:04.484 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:04.484 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:48:04.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:48:04.484 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:48:04.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:48:04.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:48:04.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:04.530 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:48:04.531 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:48:04.531 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:48:04.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:48:04.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:48:04.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:48:04.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:48:04.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:48:04.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:48:04.571 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:48:04.671 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:48:04.671 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:04.671 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:04.671 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:04.672 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:04.672 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:04.672 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:04.672 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:48:04.672 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:48:04.687 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:48:04.688 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:48:04.783 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:48:04.789 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:48:05.792 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:48:05.792 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:48:05.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:48:05.793 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:05.793 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:05.793 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:05.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:48:05.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:48:05.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:48:05.831 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:48:05.836 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:48:05.837 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:48:04.472+0000 7f4c8ce63c00 0 load: jerasure load: lrc 2026-02-25T16:48:05.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2026-02-25T16:48:05.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2026-02-25T16:48:05.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:351: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2026-02-25T16:48:05.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:353: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2026-02-25T16:48:05.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T16:48:05.838 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:48:05.839 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T16:48:05.843 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T16:48:05.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:05.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:48:05.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:05.843 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:05.844 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:05.845 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cdaa615d-ded0-40c0-944d-98f96c7ed8bb 2026-02-25T16:48:05.846 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 cdaa615d-ded0-40c0-944d-98f96c7ed8bb' 2026-02-25T16:48:05.847 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 cdaa615d-ded0-40c0-944d-98f96c7ed8bb 2026-02-25T16:48:05.847 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:05.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDFJ59pQFXQMhAAFK4H46pyo98ocIATFoQg9w== 2026-02-25T16:48:05.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDFJ59pQFXQMhAAFK4H46pyo98ocIATFoQg9w=="}' 2026-02-25T16:48:05.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cdaa615d-ded0-40c0-944d-98f96c7ed8bb -i td/test-erasure-eio/0/new.json 2026-02-25T16:48:05.957 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:05.963 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:48:05.964 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFJ59pQFXQMhAAFK4H46pyo98ocIATFoQg9w== --osd-uuid cdaa615d-ded0-40c0-944d-98f96c7ed8bb 2026-02-25T16:48:05.980 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:05.978+0000 7fbd2901d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:05.981 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:05.980+0000 7fbd2901d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:05.982 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:05.981+0000 7fbd2901d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:05.983 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:05.981+0000 7fbd2901d780 -1 bdev(0x35dec00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:05.983 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:05.981+0000 7fbd2901d780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:48:06.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:48:06.393 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:06.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:48:06.394 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:48:06.394 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:06.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:48:06.499 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:48:06.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:06.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:06.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:06.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:06.513 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:06.511+0000 7f4a63a1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:06.514 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:06.513+0000 7f4a63a1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:06.515 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:06.514+0000 7f4a63a1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:06.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:48:06.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:06.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:48:06.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:06.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:06.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:06.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:06.599 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:06.599 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:06.600 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:48:06.693 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:06.733 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:06.731+0000 7f4a63a1f780 -1 Falling back to public interface 2026-02-25T16:48:06.992 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:06.990+0000 7f4a63a1f780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:48:07.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:07.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:07.694 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:07.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:07.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:07.694 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:48:07.921 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2708934792,v1:127.0.0.1:6803/2708934792] [v2:127.0.0.1:6804/2708934792,v1:127.0.0.1:6805/2708934792] exists,up cdaa615d-ded0-40c0-944d-98f96c7ed8bb 2026-02-25T16:48:07.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:07.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:07.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:07.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:07.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:48:07.925 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:07.926 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:07.927 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=911696e5-a52c-46d6-b9d4-1c055e4b7c0c 2026-02-25T16:48:07.928 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 911696e5-a52c-46d6-b9d4-1c055e4b7c0c' 2026-02-25T16:48:07.929 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 911696e5-a52c-46d6-b9d4-1c055e4b7c0c 2026-02-25T16:48:07.929 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:07.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDHJ59plhaqNxAAIB4+DQcWwU3v759mKPcA/A== 2026-02-25T16:48:07.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDHJ59plhaqNxAAIB4+DQcWwU3v759mKPcA/A=="}' 2026-02-25T16:48:07.935 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 911696e5-a52c-46d6-b9d4-1c055e4b7c0c -i td/test-erasure-eio/1/new.json 2026-02-25T16:48:08.162 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:08.170 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:48:08.171 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHJ59plhaqNxAAIB4+DQcWwU3v759mKPcA/A== --osd-uuid 911696e5-a52c-46d6-b9d4-1c055e4b7c0c 2026-02-25T16:48:08.187 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.185+0000 7f9170a4b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:08.188 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.187+0000 7f9170a4b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:08.189 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.188+0000 7f9170a4b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:08.189 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.188+0000 7f9170a4b780 -1 bdev(0x3da4c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:08.190 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.188+0000 7f9170a4b780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:48:08.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:48:08.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:08.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:48:08.551 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:48:08.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:08.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:48:08.874 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:48:08.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:08.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:08.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:08.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:08.888 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.886+0000 7f7b759d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:08.889 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.887+0000 7f7b759d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:08.890 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:08.888+0000 7f7b759d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:09.029 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:09.028+0000 7f7b759d8780 -1 Falling back to public interface 2026-02-25T16:48:09.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:48:09.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:09.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:48:09.101 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:09.102 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:09.102 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:09.102 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:09.102 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:09.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:09.103 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:48:09.326 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:09.377 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:09.376+0000 7f7b759d8780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:48:10.327 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:10.328 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:10.328 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:10.329 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:10.329 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:10.329 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:48:10.554 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1982737908,v1:127.0.0.1:6811/1982737908] [v2:127.0.0.1:6812/1982737908,v1:127.0.0.1:6813/1982737908] exists,up 911696e5-a52c-46d6-b9d4-1c055e4b7c0c 2026-02-25T16:48:10.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:10.558 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:48:10.559 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:10.560 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:10.561 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=48def3e6-aa5c-455b-921a-734e176b8d4e 2026-02-25T16:48:10.562 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 48def3e6-aa5c-455b-921a-734e176b8d4e' 2026-02-25T16:48:10.563 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 48def3e6-aa5c-455b-921a-734e176b8d4e 2026-02-25T16:48:10.563 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:10.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKJ59pnuXOIRAAdQsYxz87D9slMS1KAGXJmQ== 2026-02-25T16:48:10.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKJ59pnuXOIRAAdQsYxz87D9slMS1KAGXJmQ=="}' 2026-02-25T16:48:10.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 48def3e6-aa5c-455b-921a-734e176b8d4e -i td/test-erasure-eio/2/new.json 2026-02-25T16:48:10.796 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:48:10.803 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:48:10.804 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKJ59pnuXOIRAAdQsYxz87D9slMS1KAGXJmQ== --osd-uuid 48def3e6-aa5c-455b-921a-734e176b8d4e 2026-02-25T16:48:10.818 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:10.817+0000 7f37ddc0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:10.820 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:10.818+0000 7f37ddc0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:10.820 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:10.819+0000 7f37ddc0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:10.821 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:10.819+0000 7f37ddc0e780 -1 bdev(0x3afcc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:10.821 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:10.819+0000 7f37ddc0e780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:48:11.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:48:11.189 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:11.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:48:11.190 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:48:11.190 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:11.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:48:11.520 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:48:11.520 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:11.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:11.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:11.521 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:11.535 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:11.533+0000 7fe2cfc42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:11.537 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:11.535+0000 7fe2cfc42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:11.537 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:11.536+0000 7fe2cfc42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:11.675 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:11.673+0000 7fe2cfc42780 -1 Falling back to public interface 2026-02-25T16:48:11.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:48:11.746 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:11.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:48:11.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:11.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:11.747 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:11.747 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:11.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:11.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:11.748 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:11.976 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:11.986 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:11.985+0000 7fe2cfc42780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:48:12.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:12.977 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:12.978 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:12.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:12.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:12.979 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:13.203 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3773134190,v1:127.0.0.1:6819/3773134190] [v2:127.0.0.1:6820/3773134190,v1:127.0.0.1:6821/3773134190] exists,up 48def3e6-aa5c-455b-921a-734e176b8d4e 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:13.208 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:13.209 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:13.210 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fdf4345c-eab9-4de0-9cd3-1be9de837b8e 2026-02-25T16:48:13.211 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 fdf4345c-eab9-4de0-9cd3-1be9de837b8e' 2026-02-25T16:48:13.212 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 fdf4345c-eab9-4de0-9cd3-1be9de837b8e 2026-02-25T16:48:13.212 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:13.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDNJ59p3m3jDBAAcHBmxK6+WQrmHmLvNWpJxg== 2026-02-25T16:48:13.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDNJ59p3m3jDBAAcHBmxK6+WQrmHmLvNWpJxg=="}' 2026-02-25T16:48:13.217 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fdf4345c-eab9-4de0-9cd3-1be9de837b8e -i td/test-erasure-eio/3/new.json 2026-02-25T16:48:13.448 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:48:13.455 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:48:13.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNJ59p3m3jDBAAcHBmxK6+WQrmHmLvNWpJxg== --osd-uuid fdf4345c-eab9-4de0-9cd3-1be9de837b8e 2026-02-25T16:48:13.472 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:13.470+0000 7effe880f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:13.473 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:13.471+0000 7effe880f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:13.474 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:13.472+0000 7effe880f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:13.474 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:13.473+0000 7effe880f780 -1 bdev(0x36ecc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:13.475 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:13.473+0000 7effe880f780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:48:13.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:48:13.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:13.829 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:48:13.830 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:48:13.830 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:14.148 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:48:14.149 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:48:14.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:14.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:14.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:14.150 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:14.163 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:14.162+0000 7fd42fc33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:14.165 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:14.163+0000 7fd42fc33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:14.165 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:14.164+0000 7fd42fc33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:14.351 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:14.349+0000 7fd42fc33780 -1 Falling back to public interface 2026-02-25T16:48:14.372 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:48:14.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:14.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:48:14.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:14.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:14.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:14.373 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:14.373 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:14.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:14.374 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:48:14.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:14.723 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:14.721+0000 7fd42fc33780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:48:15.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:15.597 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:15.598 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:15.598 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:48:15.821 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 24 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1592520906,v1:127.0.0.1:6827/1592520906] [v2:127.0.0.1:6828/1592520906,v1:127.0.0.1:6829/1592520906] exists,up fdf4345c-eab9-4de0-9cd3-1be9de837b8e 2026-02-25T16:48:15.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:15.824 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:15.825 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:15.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:48:15.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:48:15.825 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:48:15.825 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:15.826 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:15.826 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:15.826 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:48:15.826 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:48:15.826 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:48:15.866 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:48:15.869 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:48:15.871 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:48:06.733+0000 7f4a63a1f780 0 load: jerasure load: lrc 2026-02-25T16:48:15.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:355: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2026-02-25T16:48:15.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:356: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T16:48:15.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:48:15.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:48:15.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T16:48:15.871 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:48:15.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T16:48:15.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:48:15.872 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T16:48:16.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:48:16.159 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:48:16.527 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:48:16.535 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:48:17.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:48:17.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:48:17.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:48:17.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:48:17.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:48:17.538 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:48:17.539 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:48:17.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:48:17.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:48:17.539 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:48:17.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:48:17.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:48:17.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:48:17.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:48:17.630 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:48:17.630 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:48:17.855 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:48:17.856 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:17.856 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:48:17.856 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:48:17.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:48:17.856 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:17.856 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:48:17.920 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803780 2026-02-25T16:48:17.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803780 2026-02-25T16:48:17.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780' 2026-02-25T16:48:17.921 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:17.921 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:48:17.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607556 2026-02-25T16:48:17.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607556 2026-02-25T16:48:17.983 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556' 2026-02-25T16:48:17.984 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:17.984 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:48:18.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444035 2026-02-25T16:48:18.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444035 2026-02-25T16:48:18.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556 2-73014444035' 2026-02-25T16:48:18.048 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:18.048 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:48:18.110 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215108 2026-02-25T16:48:18.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215108 2026-02-25T16:48:18.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803780 1-51539607556 2-73014444035 3-103079215108' 2026-02-25T16:48:18.111 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:18.111 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803780 2026-02-25T16:48:18.111 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:18.112 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:48:18.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803780 2026-02-25T16:48:18.112 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:18.113 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803780 2026-02-25T16:48:18.114 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803780' 2026-02-25T16:48:18.114 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803780 2026-02-25T16:48:18.114 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:18.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803778 -lt 25769803780 2026-02-25T16:48:18.340 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:48:19.341 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:48:19.341 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:19.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803781 -lt 25769803780 2026-02-25T16:48:19.567 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:19.567 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607556 2026-02-25T16:48:19.567 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:19.568 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:48:19.569 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607556 2026-02-25T16:48:19.569 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:19.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607556 2026-02-25T16:48:19.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607556' 2026-02-25T16:48:19.570 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607556 2026-02-25T16:48:19.571 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:48:19.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607556 -lt 51539607556 2026-02-25T16:48:19.793 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:19.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-73014444035 2026-02-25T16:48:19.794 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:19.794 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:48:19.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-73014444035 2026-02-25T16:48:19.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:19.796 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=73014444035 2026-02-25T16:48:19.797 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 73014444035 2026-02-25T16:48:19.797 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 73014444035' 2026-02-25T16:48:19.797 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:48:20.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 73014444036 -lt 73014444035 2026-02-25T16:48:20.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:20.023 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-103079215108 2026-02-25T16:48:20.023 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:20.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:48:20.024 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-103079215108 2026-02-25T16:48:20.024 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:20.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=103079215108 2026-02-25T16:48:20.025 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 103079215108' 2026-02-25T16:48:20.026 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 103079215108 2026-02-25T16:48:20.026 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:48:20.254 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 103079215108 -lt 103079215108 2026-02-25T16:48:20.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:48:20.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:20.255 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:20.569 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:48:20.570 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:48:20.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:48:20.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:48:20.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:48:20.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:48:20.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:48:20.570 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:48:20.791 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:48:20.792 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:48:20.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:20.793 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:21.105 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:48:21.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:48:21.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:48:21.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:358: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2026-02-25T16:48:21.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T16:48:21.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T16:48:21.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T16:48:21.106 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-96095-0 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-96095-0 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-96095-0 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:21.107 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:48:21.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:21.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:48:21.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:21.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:48:21.108 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-96095-0 td/test-erasure-eio/ORIGINAL 2026-02-25T16:48:21.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-96095-0 td/test-erasure-eio 0 2026-02-25T16:48:21.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:48:21.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:48:21.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-96095-0 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=0 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:48:21.124 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-96095-0 2026-02-25T16:48:21.125 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:48:21.125 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-0 2026-02-25T16:48:21.125 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-0 2026-02-25T16:48:21.125 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:48:21.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:48:21.349 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:21.349 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:48:21.349 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:48:21.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T16:48:21.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:48:21.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=3 2026-02-25T16:48:21.350 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:48:21.350 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/3/type 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=3 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.3 2026-02-25T16:48:21.352 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:48:21.353 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:48:21.353 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:21.353 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:21.353 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:21.353 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:48:21.353 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:48:21.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:48:21.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.3 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.3 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.3 ']' 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:21.400 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.3.asok 2026-02-25T16:48:21.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:48:21.401 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-96095-0 0 2026-02-25T16:48:21.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-96095-0 2026-02-25T16:48:21.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:48:21.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:48:21.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-96095-0 2026-02-25T16:48:21.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:48:21.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:48:21.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-96095-0 td/test-erasure-eio/COPY 2026-02-25T16:48:21.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:48:21.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:48:21.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2026-02-25T16:48:21.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-96095-0 2026-02-25T16:48:21.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:48:21.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-0 2026-02-25T16:48:21.468 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-0 2026-02-25T16:48:21.468 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:48:21.694 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:48:21.694 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:21.695 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:21.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:21.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:21.696 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:22.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:22.003 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2026-02-25T16:48:22.282 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already out. 2026-02-25T16:48:22.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-96095-0 2026-02-25T16:48:22.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2026-02-25T16:48:22.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:48:22.290 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-0 2026-02-25T16:48:22.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-0 2026-02-25T16:48:22.290 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:48:22.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:48:22.518 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:22.518 INFO:tasks.workunit.client.0.trial129.stderr:0' 2026-02-25T16:48:22.518 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2026-02-25T16:48:22.519 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2026-02-25T16:48:22.800 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already in. 2026-02-25T16:48:22.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2026-02-25T16:48:22.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:22.811 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:22.812 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:22.813 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:22.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:22.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:22.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:22.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:48:22.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:48:22.814 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:48:22.815 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:48:22.815 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:48:22.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:22.816 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:48:22.816 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:48:22.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:48:22.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:48:22.817 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:48:22.826 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:22.824+0000 7f280b44e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:22.828 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:22.826+0000 7f280b44e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:22.828 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:22.827+0000 7f280b44e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:23.008 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:23.007+0000 7f280b44e780 -1 Falling back to public interface 2026-02-25T16:48:23.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:48:23.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:23.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:48:23.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:23.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:23.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:23.044 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:23.044 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:23.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:23.045 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:23.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:23.348 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:23.347+0000 7f280b44e780 -1 osd.2 32 log_to_monitors true 2026-02-25T16:48:24.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:24.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:24.271 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:24.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:24.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:24.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:24.494 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 41 up_thru 17 down_at 33 last_clean_interval [17,32) [v2:127.0.0.1:6818/1685771945,v1:127.0.0.1:6819/1685771945] [v2:127.0.0.1:6820/1685771945,v1:127.0.0.1:6821/1685771945] exists,up 48def3e6-aa5c-455b-921a-734e176b8d4e 2026-02-25T16:48:24.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:24.495 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:48:24.496 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:48:24.497 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:48:24.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:48:24.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:48:24.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:48:24.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:48:24.584 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:48:24.584 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:48:24.809 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:48:24.809 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:24.810 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:48:24.810 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:48:24.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:48:24.810 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:24.810 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:48:24.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803784 2026-02-25T16:48:24.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803784 2026-02-25T16:48:24.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784' 2026-02-25T16:48:24.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:24.876 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:48:24.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=51539607559 2026-02-25T16:48:24.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 51539607559 2026-02-25T16:48:24.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559' 2026-02-25T16:48:24.941 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:24.941 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:48:25.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659139 2026-02-25T16:48:25.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659139 2026-02-25T16:48:25.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-176093659139' 2026-02-25T16:48:25.005 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:25.005 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:48:25.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215111 2026-02-25T16:48:25.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215111 2026-02-25T16:48:25.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-176093659139 3-103079215111' 2026-02-25T16:48:25.068 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:25.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803784 2026-02-25T16:48:25.068 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:25.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:48:25.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803784 2026-02-25T16:48:25.069 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:25.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803784 2026-02-25T16:48:25.070 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-25T16:48:25.071 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803784 2026-02-25T16:48:25.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:25.299 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-25T16:48:25.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:48:26.300 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:48:26.301 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:26.529 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-25T16:48:26.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:48:27.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:48:27.531 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:27.757 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-25T16:48:27.758 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:27.758 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-51539607559 2026-02-25T16:48:27.758 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:27.759 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:48:27.759 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-51539607559 2026-02-25T16:48:27.759 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:27.760 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=51539607559 2026-02-25T16:48:27.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 51539607559' 2026-02-25T16:48:27.761 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 51539607559 2026-02-25T16:48:27.761 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:48:27.986 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 51539607560 -lt 51539607559 2026-02-25T16:48:27.987 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:27.987 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-176093659139 2026-02-25T16:48:27.987 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:27.988 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:48:27.988 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-176093659139 2026-02-25T16:48:27.988 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:27.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=176093659139 2026-02-25T16:48:27.989 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 176093659139' 2026-02-25T16:48:27.990 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 176093659139 2026-02-25T16:48:27.990 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:48:28.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 176093659139 -lt 176093659139 2026-02-25T16:48:28.215 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:28.215 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-103079215111 2026-02-25T16:48:28.216 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:28.216 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:48:28.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-103079215111 2026-02-25T16:48:28.217 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:28.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=103079215111 2026-02-25T16:48:28.218 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 103079215111' 2026-02-25T16:48:28.219 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 103079215111 2026-02-25T16:48:28.219 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:48:28.447 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 103079215112 -lt 103079215111 2026-02-25T16:48:28.448 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:48:28.448 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:28.448 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:28.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:48:28.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:48:28.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:48:28.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:48:28.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:48:28.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:48:28.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:48:28.763 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:48:28.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:48:28.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:48:28.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:28.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:29.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:48:29.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:48:29.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:48:29.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:48:29.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:48:29.303 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:48:29.569 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:48:29.577 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:48:29.878 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:48:29.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:48:29.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:48:29.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:48:29.886 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:48:29.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:29.886 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:29.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:29.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:29.887 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:29.994 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:29.995 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:48:29.996 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:48:29.996 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:48:29.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:48:29.998 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:48:29.998 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:48:29.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:29.999 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:48:30.000 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:48:30.000 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:30.001 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:30.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:48:30.002 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:48:30.012 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:48:30.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:30.013 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:30.013 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-erasure-eio 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-erasure-eio 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:30.015 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:30.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:30.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:30.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:30.016 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:30.017 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:48:30.017 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:48:30.018 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:48:30.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:48:30.019 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:48:30.019 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:48:30.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:30.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:48:30.020 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:48:30.020 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:30.021 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:30.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:48:30.022 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:48:30.023 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:48:30.023 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:30.023 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:30.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:48:30.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:48:30.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:48:30.024 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-erasure-eio 2026-02-25T16:48:30.026 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T16:48:30.026 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:30.026 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:30.026 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.96095 2026-02-25T16:48:30.027 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T16:48:30.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T16:48:30.028 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-erasure-eio 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-erasure-eio/a 2026-02-25T16:48:30.029 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-25T16:48:30.046 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T16:48:30.046 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:30.046 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:30.047 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:30.047 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:30.047 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:30.047 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:30.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T16:48:30.069 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T16:48:30.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T16:48:30.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:48:30.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:48:30.070 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T16:48:30.071 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:48:30.072 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get fsid 2026-02-25T16:48:30.116 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T16:48:30.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T16:48:30.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T16:48:30.117 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T16:48:30.118 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T16:48:30.118 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T16:48:30.118 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:48:30.118 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:48:30.118 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:30.118 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:30.119 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:30.119 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:48:30.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T16:48:30.119 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.96095/ceph-mon.a.asok config get mon_host 2026-02-25T16:48:30.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-25T16:48:30.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-erasure-eio 2026-02-25T16:48:30.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T16:48:30.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T16:48:30.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T16:48:30.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-erasure-eio/x 2026-02-25T16:48:30.163 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T16:48:30.262 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T16:48:30.262 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:30.262 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:30.262 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:30.262 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:30.262 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:30.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:30.263 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:48:30.264 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T16:48:30.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-25T16:48:30.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T16:48:30.371 INFO:tasks.workunit.client.0.trial129.stderr:pool 'rbd' already exists 2026-02-25T16:48:30.377 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-mon.a.asok 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-25T16:48:31.380 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-mon.a.asok log flush 2026-02-25T16:48:31.419 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:48:31.423 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-25T16:48:31.425 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:48:30.061+0000 7f1c26e58c00 0 load: jerasure load: lrc 2026-02-25T16:48:31.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2026-02-25T16:48:31.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2026-02-25T16:48:31.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2026-02-25T16:48:31.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2026-02-25T16:48:31.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-25T16:48:31.426 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-25T16:48:31.426 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-25T16:48:31.427 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-25T16:48:31.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:31.431 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-25T16:48:31.432 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:31.433 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ce56bdd0-2403-4936-b60c-07c1faea7b5b 2026-02-25T16:48:31.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ce56bdd0-2403-4936-b60c-07c1faea7b5b' 2026-02-25T16:48:31.435 INFO:tasks.workunit.client.0.trial129.stdout:add osd0 ce56bdd0-2403-4936-b60c-07c1faea7b5b 2026-02-25T16:48:31.435 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:31.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDfJ59ptfA2GhAAUbwlghbaYwjd1G250peumg== 2026-02-25T16:48:31.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDfJ59ptfA2GhAAUbwlghbaYwjd1G250peumg=="}' 2026-02-25T16:48:31.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ce56bdd0-2403-4936-b60c-07c1faea7b5b -i td/test-erasure-eio/0/new.json 2026-02-25T16:48:31.544 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:31.550 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-25T16:48:31.551 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfJ59ptfA2GhAAUbwlghbaYwjd1G250peumg== --osd-uuid ce56bdd0-2403-4936-b60c-07c1faea7b5b 2026-02-25T16:48:31.566 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:31.564+0000 7fe6b4e1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:31.567 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:31.566+0000 7fe6b4e1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:31.568 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:31.567+0000 7fe6b4e1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:31.569 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:31.567+0000 7fe6b4e1f780 -1 bdev(0x3260c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:31.569 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:31.567+0000 7fe6b4e1f780 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-25T16:48:31.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-25T16:48:31.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:31.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T16:48:31.974 INFO:tasks.workunit.client.0.trial129.stdout:adding osd0 key to auth repository 2026-02-25T16:48:31.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:32.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T16:48:32.076 INFO:tasks.workunit.client.0.trial129.stdout:start osd.0 2026-02-25T16:48:32.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:32.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:32.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:32.077 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:32.091 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:32.090+0000 7fe27900f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:32.093 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:32.092+0000 7fe27900f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:32.094 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:32.093+0000 7fe27900f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:32.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T16:48:32.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:32.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T16:48:32.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:32.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:32.173 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:32.173 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:32.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:32.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:32.174 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:48:32.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:32.301 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:32.300+0000 7fe27900f780 -1 Falling back to public interface 2026-02-25T16:48:32.579 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:32.577+0000 7fe27900f780 -1 osd.0 0 log_to_monitors true 2026-02-25T16:48:33.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:33.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:33.271 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:33.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:33.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:33.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T16:48:33.498 INFO:tasks.workunit.client.0.trial129.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/462621763,v1:127.0.0.1:6803/462621763] [v2:127.0.0.1:6804/462621763,v1:127.0.0.1:6805/462621763] exists,up ce56bdd0-2403-4936-b60c-07c1faea7b5b 2026-02-25T16:48:33.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:33.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:33.504 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:33.505 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:33.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:33.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-25T16:48:33.506 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:33.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4a818d37-6589-4df3-a408-444a2083f19a 2026-02-25T16:48:33.506 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4a818d37-6589-4df3-a408-444a2083f19a' 2026-02-25T16:48:33.506 INFO:tasks.workunit.client.0.trial129.stdout:add osd1 4a818d37-6589-4df3-a408-444a2083f19a 2026-02-25T16:48:33.506 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:33.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDhJ59p/bx3HhAAtjextRLly+sWhDgPZLHkeQ== 2026-02-25T16:48:33.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDhJ59p/bx3HhAAtjextRLly+sWhDgPZLHkeQ=="}' 2026-02-25T16:48:33.512 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4a818d37-6589-4df3-a408-444a2083f19a -i td/test-erasure-eio/1/new.json 2026-02-25T16:48:33.744 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:33.753 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-25T16:48:33.754 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhJ59p/bx3HhAAtjextRLly+sWhDgPZLHkeQ== --osd-uuid 4a818d37-6589-4df3-a408-444a2083f19a 2026-02-25T16:48:33.769 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:33.768+0000 7f2c23ffb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:33.771 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:33.769+0000 7f2c23ffb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:33.771 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:33.770+0000 7f2c23ffb780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:33.772 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:33.770+0000 7f2c23ffb780 -1 bdev(0x364ac00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:33.772 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:33.770+0000 7f2c23ffb780 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-25T16:48:34.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-25T16:48:34.117 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:34.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T16:48:34.118 INFO:tasks.workunit.client.0.trial129.stdout:adding osd1 key to auth repository 2026-02-25T16:48:34.118 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:34.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T16:48:34.435 INFO:tasks.workunit.client.0.trial129.stdout:start osd.1 2026-02-25T16:48:34.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:34.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:34.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:34.436 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:34.449 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:34.447+0000 7fce63056780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:34.450 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:34.448+0000 7fce63056780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:34.451 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:34.449+0000 7fce63056780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:34.595 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:34.594+0000 7fce63056780 -1 Falling back to public interface 2026-02-25T16:48:34.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T16:48:34.659 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:34.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T16:48:34.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:34.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:34.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:34.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:34.660 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:34.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:34.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:48:34.890 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:34.937 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:34.935+0000 7fce63056780 -1 osd.1 0 log_to_monitors true 2026-02-25T16:48:35.891 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:35.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:35.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:35.892 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:35.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:35.892 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T16:48:36.115 INFO:tasks.workunit.client.0.trial129.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3832209225,v1:127.0.0.1:6811/3832209225] [v2:127.0.0.1:6812/3832209225,v1:127.0.0.1:6813/3832209225] exists,up 4a818d37-6589-4df3-a408-444a2083f19a 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T16:48:36.119 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:36.120 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:36.121 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=89f3cee1-5ebe-4f35-b3b9-09d7245bf66e 2026-02-25T16:48:36.122 INFO:tasks.workunit.client.0.trial129.stdout:add osd2 89f3cee1-5ebe-4f35-b3b9-09d7245bf66e 2026-02-25T16:48:36.123 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 89f3cee1-5ebe-4f35-b3b9-09d7245bf66e' 2026-02-25T16:48:36.123 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:36.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDkJ59pXgihBxAA240x+D6Vjb5L6x1A0WcYKg== 2026-02-25T16:48:36.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDkJ59pXgihBxAA240x+D6Vjb5L6x1A0WcYKg=="}' 2026-02-25T16:48:36.129 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 89f3cee1-5ebe-4f35-b3b9-09d7245bf66e -i td/test-erasure-eio/2/new.json 2026-02-25T16:48:36.357 INFO:tasks.workunit.client.0.trial129.stdout:2 2026-02-25T16:48:36.364 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-25T16:48:36.365 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkJ59pXgihBxAA240x+D6Vjb5L6x1A0WcYKg== --osd-uuid 89f3cee1-5ebe-4f35-b3b9-09d7245bf66e 2026-02-25T16:48:36.380 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:36.379+0000 7ff47c256780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:36.382 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:36.380+0000 7ff47c256780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:36.383 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:36.381+0000 7ff47c256780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:36.383 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:36.381+0000 7ff47c256780 -1 bdev(0x2b62c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:36.383 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:36.382+0000 7ff47c256780 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-25T16:48:36.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-25T16:48:36.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:36.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T16:48:36.731 INFO:tasks.workunit.client.0.trial129.stdout:adding osd2 key to auth repository 2026-02-25T16:48:36.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:37.050 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T16:48:37.051 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:48:37.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:37.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:37.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:37.052 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:37.066 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:37.064+0000 7f7b57c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:37.067 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:37.065+0000 7f7b57c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:37.068 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:37.066+0000 7f7b57c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:37.200 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:37.198+0000 7f7b57c56780 -1 Falling back to public interface 2026-02-25T16:48:37.276 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T16:48:37.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:37.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:48:37.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:37.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:37.277 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:37.277 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:37.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:37.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:37.278 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:37.500 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:37.559 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:37.557+0000 7f7b57c56780 -1 osd.2 0 log_to_monitors true 2026-02-25T16:48:38.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:38.501 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:38.502 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:38.502 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:38.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:38.503 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:38.725 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1964150120,v1:127.0.0.1:6819/1964150120] [v2:127.0.0.1:6820/1964150120,v1:127.0.0.1:6821/1964150120] exists,up 89f3cee1-5ebe-4f35-b3b9-09d7245bf66e 2026-02-25T16:48:38.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T16:48:38.730 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:38.731 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:38.732 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c244a5c3-2454-4396-aa9b-42d0d056ad17 2026-02-25T16:48:38.733 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 c244a5c3-2454-4396-aa9b-42d0d056ad17' 2026-02-25T16:48:38.734 INFO:tasks.workunit.client.0.trial129.stdout:add osd3 c244a5c3-2454-4396-aa9b-42d0d056ad17 2026-02-25T16:48:38.734 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T16:48:38.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDmJ59pIZsFLBAAddr6ByGqlAxzUdJF7EfNMw== 2026-02-25T16:48:38.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDmJ59pIZsFLBAAddr6ByGqlAxzUdJF7EfNMw=="}' 2026-02-25T16:48:38.740 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c244a5c3-2454-4396-aa9b-42d0d056ad17 -i td/test-erasure-eio/3/new.json 2026-02-25T16:48:38.966 INFO:tasks.workunit.client.0.trial129.stdout:3 2026-02-25T16:48:38.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-25T16:48:38.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmJ59pIZsFLBAAddr6ByGqlAxzUdJF7EfNMw== --osd-uuid c244a5c3-2454-4396-aa9b-42d0d056ad17 2026-02-25T16:48:38.989 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:38.987+0000 7f8a7b02d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:38.990 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:38.989+0000 7f8a7b02d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:38.991 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:38.989+0000 7f8a7b02d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:38.992 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:38.990+0000 7f8a7b02d780 -1 bdev(0x31cac00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-25T16:48:38.992 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:38.990+0000 7f8a7b02d780 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-25T16:48:39.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-25T16:48:39.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T16:48:39.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T16:48:39.336 INFO:tasks.workunit.client.0.trial129.stdout:adding osd3 key to auth repository 2026-02-25T16:48:39.336 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T16:48:39.647 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T16:48:39.647 INFO:tasks.workunit.client.0.trial129.stdout:start osd.3 2026-02-25T16:48:39.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:39.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T16:48:39.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T16:48:39.648 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T16:48:39.660 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:39.659+0000 7fd38862b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:39.661 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:39.660+0000 7fd38862b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:39.663 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:39.661+0000 7fd38862b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:39.842 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:39.841+0000 7fd38862b780 -1 Falling back to public interface 2026-02-25T16:48:39.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T16:48:39.874 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:39.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T16:48:39.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:39.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:39.875 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:39.875 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:39.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:39.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:39.876 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:48:40.098 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:40.224 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:40.222+0000 7fd38862b780 -1 osd.3 0 log_to_monitors true 2026-02-25T16:48:41.099 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:41.099 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:41.099 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:41.099 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:41.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:41.100 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T16:48:41.327 INFO:tasks.workunit.client.0.trial129.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2150211119,v1:127.0.0.1:6827/2150211119] [v2:127.0.0.1:6828/2150211119,v1:127.0.0.1:6829/2150211119] exists,up c244a5c3-2454-4396-aa9b-42d0d056ad17 2026-02-25T16:48:41.329 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:41.329 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:41.329 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:41.329 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.0.asok 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-25T16:48:41.330 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.0.asok log flush 2026-02-25T16:48:41.372 INFO:tasks.workunit.client.0.trial129.stdout:{} 2026-02-25T16:48:41.376 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-25T16:48:41.378 INFO:tasks.workunit.client.0.trial129.stdout:2026-02-25T16:48:32.302+0000 7fe27900f780 0 load: jerasure load: lrc 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:370: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-25T16:48:41.379 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-25T16:48:41.660 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-25T16:48:41.661 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-25T16:48:42.033 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' already exists 2026-02-25T16:48:42.041 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T16:48:43.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-25T16:48:43.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:48:43.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:48:43.043 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:48:43.044 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:48:43.044 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:48:43.044 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:48:43.044 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:48:43.044 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:48:43.044 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:48:43.135 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:48:43.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:48:43.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:48:43.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:48:43.136 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:48:43.136 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:48:43.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:48:43.369 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:43.369 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:48:43.369 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:48:43.369 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:48:43.370 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:43.370 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:48:43.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803781 2026-02-25T16:48:43.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803781 2026-02-25T16:48:43.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781' 2026-02-25T16:48:43.434 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:43.434 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:48:43.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-25T16:48:43.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-25T16:48:43.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574852' 2026-02-25T16:48:43.499 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:43.499 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:48:43.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378627 2026-02-25T16:48:43.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378627 2026-02-25T16:48:43.564 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574852 2-81604378627' 2026-02-25T16:48:43.565 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:43.565 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:48:43.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116996 2026-02-25T16:48:43.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116996 2026-02-25T16:48:43.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803781 1-55834574852 2-81604378627 3-115964116996' 2026-02-25T16:48:43.629 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:43.629 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803781 2026-02-25T16:48:43.629 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:43.631 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:48:43.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803781 2026-02-25T16:48:43.632 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:43.632 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803781 2026-02-25T16:48:43.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2026-02-25T16:48:43.633 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803781 2026-02-25T16:48:43.633 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:43.853 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803779 -lt 25769803781 2026-02-25T16:48:43.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:48:44.854 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:48:44.855 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:45.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803782 -lt 25769803781 2026-02-25T16:48:45.082 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:45.083 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574852 2026-02-25T16:48:45.083 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:45.083 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:48:45.084 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574852 2026-02-25T16:48:45.084 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:45.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574852 2026-02-25T16:48:45.085 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574852' 2026-02-25T16:48:45.085 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574852 2026-02-25T16:48:45.086 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:48:45.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-25T16:48:45.309 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:45.310 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-81604378627 2026-02-25T16:48:45.310 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:45.311 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:48:45.311 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-81604378627 2026-02-25T16:48:45.311 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:45.312 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378627 2026-02-25T16:48:45.313 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 81604378627 2026-02-25T16:48:45.313 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 81604378627' 2026-02-25T16:48:45.313 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:48:45.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378628 -lt 81604378627 2026-02-25T16:48:45.536 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:45.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116996 2026-02-25T16:48:45.537 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:45.537 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:48:45.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116996 2026-02-25T16:48:45.538 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:45.539 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116996 2026-02-25T16:48:45.540 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116996 2026-02-25T16:48:45.540 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116996' 2026-02-25T16:48:45.540 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:48:45.763 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964116996 -lt 115964116996 2026-02-25T16:48:45.764 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:48:45.764 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:45.764 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:46.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:48:46.076 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:48:46.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:48:46.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:48:46.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:48:46.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:48:46.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:48:46.077 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:48:46.306 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:48:46.306 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:48:46.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:46.307 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:46.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:48:46.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:48:46.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:48:46.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2026-02-25T16:48:46.616 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-96095-1 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-96095-1 2026-02-25T16:48:46.617 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-96095-1 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-25T16:48:46.618 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-96095-1 td/test-erasure-eio/ORIGINAL 2026-02-25T16:48:46.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-96095-1 td/test-erasure-eio 1 2026-02-25T16:48:46.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2480: inject_eio: local pooltype=ec 2026-02-25T16:48:46.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: shift 2026-02-25T16:48:46.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: local which=data 2026-02-25T16:48:46.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: shift 2026-02-25T16:48:46.633 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: local poolname=pool-jerasure 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: shift 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: local objname=obj-eio-96095-1 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: shift 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: local dir=td/test-erasure-eio 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: shift 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: local shard_id=1 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: shift 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: get_osds pool-jerasure obj-eio-96095-1 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-1 2026-02-25T16:48:46.634 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-1 2026-02-25T16:48:46.635 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:48:46.859 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:48:46.859 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:46.859 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:48:46.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:48:46.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: initial_osds=('3' '1' '2') 2026-02-25T16:48:46.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2493: inject_eio: local -a initial_osds 2026-02-25T16:48:46.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local osd_id=1 2026-02-25T16:48:46.860 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: '[' ec '!=' ec ']' 2026-02-25T16:48:46.860 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: cat td/test-erasure-eio/1/type 2026-02-25T16:48:46.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2498: inject_eio: type=bluestore 2026-02-25T16:48:46.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-25T16:48:46.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-25T16:48:46.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-25T16:48:46.862 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:48:46.863 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.96095/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-25T16:48:46.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-25T16:48:46.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: local loop=0 2026-02-25T16:48:46.909 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: grep -q Invalid 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: get_asok_path osd.1 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.96095/ceph-osd.1.asok 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: CEPH_ARGS= 2026-02-25T16:48:46.910 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: ceph --admin-daemon /tmp/ceph-asok.96095/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-96095-1 1 2026-02-25T16:48:46.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-96095-1 2026-02-25T16:48:46.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-25T16:48:46.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-25T16:48:46.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-96095-1 2026-02-25T16:48:46.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-25T16:48:46.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-25T16:48:46.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-96095-1 td/test-erasure-eio/COPY 2026-02-25T16:48:46.973 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-25T16:48:46.974 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-25T16:48:46.975 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2026-02-25T16:48:46.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-96095-1 2026-02-25T16:48:46.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:48:46.976 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-1 2026-02-25T16:48:46.977 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-1 2026-02-25T16:48:46.977 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr:2' 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:47.202 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:47.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:47.203 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:47.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:47.510 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2026-02-25T16:48:47.786 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already out. 2026-02-25T16:48:47.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-96095-1 2026-02-25T16:48:47.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2026-02-25T16:48:47.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=pool-jerasure 2026-02-25T16:48:47.795 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=obj-eio-96095-1 2026-02-25T16:48:47.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map pool-jerasure obj-eio-96095-1 2026-02-25T16:48:47.795 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T16:48:48.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-25T16:48:48.023 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:48.023 INFO:tasks.workunit.client.0.trial129.stderr:0' 2026-02-25T16:48:48.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 0 2026-02-25T16:48:48.023 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2026-02-25T16:48:48.257 INFO:tasks.workunit.client.0.trial129.stderr:osd.2 is already in. 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/test-erasure-eio 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T16:48:48.268 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:48.269 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-25T16:48:48.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-25T16:48:48.271 INFO:tasks.workunit.client.0.trial129.stdout:start osd.2 2026-02-25T16:48:48.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8230a16a-1dbc-4385-a9e8-3542bf145d17 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.96095/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T16:48:48.272 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-25T16:48:48.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-25T16:48:48.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T16:48:48.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T16:48:48.272 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T16:48:48.281 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:48.280+0000 7ff980834780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:48.283 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:48.281+0000 7ff980834780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:48.283 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:48.282+0000 7ff980834780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T16:48:48.449 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:48.447+0000 7ff980834780 -1 Falling back to public interface 2026-02-25T16:48:48.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-25T16:48:48.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T16:48:48.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T16:48:48.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T16:48:48.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T16:48:48.497 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:48.498 INFO:tasks.workunit.client.0.trial129.stdout:0 2026-02-25T16:48:48.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T16:48:48.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:48.498 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:48.726 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T16:48:48.768 INFO:tasks.workunit.client.0.trial129.stderr:2026-02-25T16:48:48.766+0000 7ff980834780 -1 osd.2 35 log_to_monitors true 2026-02-25T16:48:49.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T16:48:49.728 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T16:48:49.728 INFO:tasks.workunit.client.0.trial129.stdout:1 2026-02-25T16:48:49.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T16:48:49.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T16:48:49.729 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T16:48:49.955 INFO:tasks.workunit.client.0.trial129.stdout:osd.2 up in weight 1 up_from 44 up_thru 19 down_at 36 last_clean_interval [19,35) [v2:127.0.0.1:6818/1485721100,v1:127.0.0.1:6819/1485721100] [v2:127.0.0.1:6820/1485721100,v1:127.0.0.1:6821/1485721100] exists,up 89f3cee1-5ebe-4f35-b3b9-09d7245bf66e 2026-02-25T16:48:49.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T16:48:49.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T16:48:49.957 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T16:48:49.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2026-02-25T16:48:49.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T16:48:49.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T16:48:49.958 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T16:48:49.958 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T16:48:49.958 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T16:48:49.959 INFO:tasks.workunit.client.0.trial129.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T16:48:49.959 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T16:48:49.959 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T16:48:49.959 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T16:48:50.046 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T16:48:50.046 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T16:48:50.046 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T16:48:50.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T16:48:50.047 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T16:48:50.047 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T16:48:50.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T16:48:50.270 INFO:tasks.workunit.client.0.trial129.stderr:1 2026-02-25T16:48:50.270 INFO:tasks.workunit.client.0.trial129.stderr:2 2026-02-25T16:48:50.270 INFO:tasks.workunit.client.0.trial129.stderr:3' 2026-02-25T16:48:50.270 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T16:48:50.271 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:50.271 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T16:48:50.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=25769803785 2026-02-25T16:48:50.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 25769803785 2026-02-25T16:48:50.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785' 2026-02-25T16:48:50.335 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:50.335 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T16:48:50.397 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-25T16:48:50.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-25T16:48:50.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855' 2026-02-25T16:48:50.398 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:50.398 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T16:48:50.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561027 2026-02-25T16:48:50.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561027 2026-02-25T16:48:50.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-188978561027' 2026-02-25T16:48:50.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T16:48:50.464 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T16:48:50.526 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=115964116999 2026-02-25T16:48:50.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 115964116999 2026-02-25T16:48:50.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-188978561027 3-115964116999' 2026-02-25T16:48:50.527 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:50.527 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-25769803785 2026-02-25T16:48:50.527 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:50.528 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T16:48:50.528 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-25769803785 2026-02-25T16:48:50.529 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:50.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=25769803785 2026-02-25T16:48:50.530 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-25T16:48:50.530 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.0 seq 25769803785 2026-02-25T16:48:50.530 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:50.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803785 2026-02-25T16:48:50.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:48:51.761 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T16:48:51.762 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:51.990 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803783 -lt 25769803785 2026-02-25T16:48:51.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T16:48:52.991 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T16:48:52.992 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T16:48:53.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-25T16:48:53.220 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:53.221 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-55834574855 2026-02-25T16:48:53.221 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:53.222 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T16:48:53.222 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-55834574855 2026-02-25T16:48:53.222 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:53.223 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574855 2026-02-25T16:48:53.224 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 55834574855' 2026-02-25T16:48:53.224 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.1 seq 55834574855 2026-02-25T16:48:53.224 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T16:48:53.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574856 -lt 55834574855 2026-02-25T16:48:53.448 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:53.449 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-188978561027 2026-02-25T16:48:53.449 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:53.449 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T16:48:53.450 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-188978561027 2026-02-25T16:48:53.450 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:53.451 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=188978561027 2026-02-25T16:48:53.451 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 188978561027' 2026-02-25T16:48:53.452 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.2 seq 188978561027 2026-02-25T16:48:53.452 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T16:48:53.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 188978561027 -lt 188978561027 2026-02-25T16:48:53.679 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T16:48:53.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-115964116999 2026-02-25T16:48:53.680 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T16:48:53.680 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T16:48:53.681 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-115964116999 2026-02-25T16:48:53.681 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T16:48:53.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=115964116999 2026-02-25T16:48:53.682 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 115964116999' 2026-02-25T16:48:53.683 INFO:tasks.workunit.client.0.trial129.stdout:waiting osd.3 seq 115964116999 2026-02-25T16:48:53.683 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T16:48:53.911 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 115964117000 -lt 115964116999 2026-02-25T16:48:53.912 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T16:48:53.912 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:53.912 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:54.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-25T16:48:54.227 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T16:48:54.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T16:48:54.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T16:48:54.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T16:48:54.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T16:48:54.227 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T16:48:54.228 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T16:48:54.447 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-25T16:48:54.448 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T16:48:54.448 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T16:48:54.448 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T16:48:54.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-25T16:48:54.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T16:48:54.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T16:48:54.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:374: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-25T16:48:54.762 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-25T16:48:54.763 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-25T16:48:55.034 INFO:tasks.workunit.client.0.trial129.stderr:pool 'pool-jerasure' does not exist 2026-02-25T16:48:55.042 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-25T16:48:55.323 INFO:tasks.workunit.client.0.trial129.stderr:erasure-code-profile myprofile does not exist 2026-02-25T16:48:55.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-25T16:48:55.331 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:48:55.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T16:48:55.332 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:48:55.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:55.332 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:55.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:55.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:55.333 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:55.440 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:55.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:48:55.441 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:48:55.441 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:48:55.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:48:55.442 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:48:55.443 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:48:55.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:55.443 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:48:55.444 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:48:55.444 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:55.445 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:55.446 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T16:48:55.446 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:48:55.456 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:48:55.456 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:55.456 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:55.456 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/test-erasure-eio 0 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-erasure-eio 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:48:55.458 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:48:55.459 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:48:55.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:48:55.460 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:48:55.460 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:48:55.461 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:48:55.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:48:55.462 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:48:55.462 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:48:55.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:55.463 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:48:55.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:48:55.464 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:48:55.464 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:55.465 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T16:48:55.466 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-erasure-eio 2026-02-25T16:48:55.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T16:48:55.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T16:48:55.467 INFO:tasks.workunit.client.0.trial129.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.96095 2026-02-25T16:48:55.467 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.96095 2026-02-25T16:48:55.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T16:48:55.469 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T16:48:55.470 INFO:tasks.workunit.client.0.trial129.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T16:48:55.470 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T16:48:55.470 DEBUG:teuthology.orchestra.run.trial129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T16:48:55.497 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2026-02-25T16:48:55.498 DEBUG:teuthology.orchestra.run.trial129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-25T16:48:55.860 DEBUG:teuthology.parallel:result is None 2026-02-25T16:48:55.860 DEBUG:teuthology.orchestra.run.trial129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T16:48:55.885 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T16:48:55.886 DEBUG:teuthology.orchestra.run.trial129:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T16:48:55.943 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-25T16:48:55.943 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T16:48:55.957 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T16:48:55.957 DEBUG:teuthology.orchestra.run.trial129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T16:48:56.102 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T16:48:56.114 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T16:48:56.114 DEBUG:teuthology.orchestra.run.trial129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T16:48:56.130 INFO:teuthology.orchestra.run.trial129.stderr:bash: line 1: ntpq: command not found 2026-02-25T16:48:56.300 INFO:teuthology.orchestra.run.trial129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T16:48:56.300 INFO:teuthology.orchestra.run.trial129.stdout:=============================================================================== 2026-02-25T16:48:56.300 INFO:teuthology.orchestra.run.trial129.stdout:^* 69-172-133-130.planet.net 2 6 377 7 +12us[ +44us] +/- 3895us 2026-02-25T16:48:56.300 INFO:teuthology.orchestra.run.trial129.stdout:^- ip229.ip-51-81-226.us 2 6 377 3 +1226us[+1226us] +/- 92ms 2026-02-25T16:48:56.300 INFO:teuthology.orchestra.run.trial129.stdout:^- connected.by.freedominte> 2 6 377 4 +4848ns[+4848ns] +/- 42ms 2026-02-25T16:48:56.300 INFO:teuthology.orchestra.run.trial129.stdout:^- 204.2.134.173 3 6 377 2 -1252us[-1252us] +/- 91ms 2026-02-25T16:48:56.301 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T16:48:56.311 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T16:48:56.312 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T16:48:56.328 DEBUG:teuthology.orchestra.run.trial129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T16:48:56.392 DEBUG:teuthology.orchestra.run.trial129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T16:48:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T16:48:56.419 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T16:48:56.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T16:48:56.448 INFO:teuthology.task.internal:Duration was 1019.622203 seconds 2026-02-25T16:48:56.448 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T16:48:56.459 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T16:48:56.459 DEBUG:teuthology.orchestra.run.trial129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T16:48:56.496 INFO:teuthology.orchestra.run.trial129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T16:48:56.811 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T16:48:56.811 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial129.front.sepia.ceph.com 2026-02-25T16:48:56.811 DEBUG:teuthology.orchestra.run.trial129:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T16:48:56.868 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T16:48:56.868 DEBUG:teuthology.orchestra.run.trial129:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T16:48:56.966 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T16:48:56.966 DEBUG:teuthology.orchestra.run.trial129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T16:48:56.988 INFO:teuthology.orchestra.run.trial129.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T16:48:56.988 INFO:teuthology.orchestra.run.trial129.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T16:48:56.988 INFO:teuthology.orchestra.run.trial129.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T16:48:56.989 INFO:teuthology.orchestra.run.trial129.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T16:48:56.989 INFO:teuthology.orchestra.run.trial129.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T16:48:56.998 INFO:teuthology.orchestra.run.trial129.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T16:48:57.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T16:48:57.012 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T16:48:57.012 DEBUG:teuthology.orchestra.run.trial129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T16:48:57.064 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T16:48:57.080 DEBUG:teuthology.orchestra.run.trial129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:57.128 INFO:teuthology.orchestra.run.trial129.stdout:kernel.core_pattern = core 2026-02-25T16:48:57.141 DEBUG:teuthology.orchestra.run.trial129:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T16:48:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T16:48:57.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T16:48:57.213 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T16:48:57.215 DEBUG:teuthology.misc:Transferring archived files from trial129:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_16:24:30-rados:standalone-wip-mon-colocate-distro-default-trial/70446/remote/trial129 2026-02-25T16:48:57.217 DEBUG:teuthology.orchestra.run.trial129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T16:48:57.279 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T16:48:57.279 DEBUG:teuthology.orchestra.run.trial129:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T16:48:57.323 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T16:48:57.340 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T16:48:57.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T16:48:57.355 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T16:48:57.356 DEBUG:teuthology.orchestra.run.trial129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T16:48:57.378 INFO:teuthology.orchestra.run.trial129.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 16:48 /home/ubuntu/cephtest 2026-02-25T16:48:57.380 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T16:48:57.396 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T16:48:57.433 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} duration: 1019.62220287323 flavor: default owner: scheduled_emzhang@soko04.front.sepia.ceph.com success: true 2026-02-25T16:48:57.433 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T16:48:57.550 INFO:teuthology.run:pass